Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 08:11:05.453812 28279 test_util.cc:268] Using random seed: 1122198722
I20240503 08:11:05.465600 28279 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:05.465689 28279 ts_itest-base.cc:113] --------------
I20240503 08:11:05.465739 28279 ts_itest-base.cc:114] 5 tablet servers
I20240503 08:11:05.465808 28279 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:11:05.465885 28279 ts_itest-base.cc:116] --------------
2024-05-03T08:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:05Z Disabled control of system clock
I20240503 08:11:05.491708 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.157.254:34507
--webserver_interface=127.27.157.254
--webserver_port=0
--builtin_ntp_servers=127.27.157.212:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.157.254:34507
--enable_range_replica_placement=false with env {}
W20240503 08:11:05.832984 28291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:05.833314 28291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:05.833420 28291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:05.841109 28291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:05.841235 28291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:05.841287 28291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:05.841347 28291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:05.850713 28291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.157.254:34507
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.157.254:34507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.157.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:05.852094 28291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:05.854188 28291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:05.866117 28297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:05.866678 28296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:05.867555 28299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:05.868362 28291 server_base.cc:1034] running on GCE node
I20240503 08:11:05.869299 28291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:05.870867 28291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:05.872141 28291 hybrid_clock.cc:648] HybridClock initialized: now 1714723865872079 us; error 34 us; skew 500 ppm
I20240503 08:11:05.872521 28291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:05.875068 28291 webserver.cc:460] Webserver started at http://127.27.157.254:34737/ using document root <none> and password file <none>
I20240503 08:11:05.875880 28291 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:05.876025 28291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:05.876345 28291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:05.879312 28291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0ffa2bb078774779b0683355794c0526"
format_stamp: "Formatted at 2024-05-03 08:11:05 on dist-test-slave-dj8w"
I20240503 08:11:05.880177 28291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0ffa2bb078774779b0683355794c0526"
format_stamp: "Formatted at 2024-05-03 08:11:05 on dist-test-slave-dj8w"
I20240503 08:11:05.885879 28291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240503 08:11:05.889760 28305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:05.891264 28291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:05.891507 28291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0ffa2bb078774779b0683355794c0526"
format_stamp: "Formatted at 2024-05-03 08:11:05 on dist-test-slave-dj8w"
I20240503 08:11:05.891719 28291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:05.910812 28291 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:05.911746 28291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:05.912065 28291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:05.931489 28291 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.254:34507
I20240503 08:11:05.931529 28356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.254:34507 every 8 connection(s)
I20240503 08:11:05.932967 28291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:05.936764 28357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:05.939934 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 28291
I20240503 08:11:05.940243 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:05.948956 28357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526: Bootstrap starting.
I20240503 08:11:05.952087 28357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:05.953122 28357 log.cc:824] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:05.956085 28357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526: No bootstrap required, opened a new log
I20240503 08:11:05.965322 28357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa2bb078774779b0683355794c0526" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 34507 } }
I20240503 08:11:05.965751 28357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:05.965881 28357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ffa2bb078774779b0683355794c0526, State: Initialized, Role: FOLLOWER
I20240503 08:11:05.966531 28357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa2bb078774779b0683355794c0526" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 34507 } }
I20240503 08:11:05.966812 28357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:05.966964 28357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:05.967150 28357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:05.970284 28357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa2bb078774779b0683355794c0526" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 34507 } }
I20240503 08:11:05.970796 28357 leader_election.cc:302] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ffa2bb078774779b0683355794c0526; no voters:
I20240503 08:11:05.971465 28357 leader_election.cc:288] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:05.971626 28362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:05.973054 28362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [term 1 LEADER]: Becoming Leader. State: Replica: 0ffa2bb078774779b0683355794c0526, State: Running, Role: LEADER
I20240503 08:11:05.973608 28362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa2bb078774779b0683355794c0526" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 34507 } }
I20240503 08:11:05.973740 28357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:05.976259 28364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ffa2bb078774779b0683355794c0526. Latest consensus state: current_term: 1 leader_uuid: "0ffa2bb078774779b0683355794c0526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa2bb078774779b0683355794c0526" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 34507 } } }
I20240503 08:11:05.976640 28364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:05.976348 28363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ffa2bb078774779b0683355794c0526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ffa2bb078774779b0683355794c0526" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 34507 } } }
I20240503 08:11:05.977007 28363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:05.980669 28370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:05.986341 28370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:05.994927 28370 catalog_manager.cc:1347] Generated new cluster ID: c22a14575cbd4d2a83f2bb262b888004
I20240503 08:11:05.995069 28370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:06.011734 28370 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:06.013197 28370 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:06.025861 28370 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0ffa2bb078774779b0683355794c0526: Generated new TSK 0
I20240503 08:11:06.026863 28370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:06.037825 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.193:0
--local_ip_for_outbound_sockets=127.27.157.193
--webserver_interface=127.27.157.193
--webserver_port=0
--tserver_master_addrs=127.27.157.254:34507
--builtin_ntp_servers=127.27.157.212:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 08:11:06.380391 28381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:06.380734 28381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:06.380856 28381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:06.388556 28381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:06.388748 28381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.193
I20240503 08:11:06.398413 28381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.157.193
--webserver_port=0
--tserver_master_addrs=127.27.157.254:34507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.193
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:06.399843 28381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:06.401551 28381 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:06.414510 28387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.415174 28389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.415861 28386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:06.416574 28381 server_base.cc:1034] running on GCE node
I20240503 08:11:06.417266 28381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:06.418269 28381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:06.419466 28381 hybrid_clock.cc:648] HybridClock initialized: now 1714723866419390 us; error 46 us; skew 500 ppm
I20240503 08:11:06.419790 28381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:06.422302 28381 webserver.cc:460] Webserver started at http://127.27.157.193:45629/ using document root <none> and password file <none>
I20240503 08:11:06.423063 28381 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:06.423195 28381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:06.423687 28381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:06.430116 28381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a7cb0e6105c149db99971899ca8272e2"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-dj8w"
I20240503 08:11:06.430918 28381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a7cb0e6105c149db99971899ca8272e2"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-dj8w"
I20240503 08:11:06.436688 28381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240503 08:11:06.440372 28395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:06.441946 28381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:11:06.442123 28381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a7cb0e6105c149db99971899ca8272e2"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-dj8w"
I20240503 08:11:06.442313 28381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:06.455652 28381 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:06.456499 28381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:06.456800 28381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:06.457762 28381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:06.459301 28381 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:06.459425 28381 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:06.459558 28381 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:06.459611 28381 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:06.492971 28381 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.193:37333
I20240503 08:11:06.493029 28507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.193:37333 every 8 connection(s)
I20240503 08:11:06.494325 28381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:06.499208 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 28381
I20240503 08:11:06.499450 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:06.503356 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.194:0
--local_ip_for_outbound_sockets=127.27.157.194
--webserver_interface=127.27.157.194
--webserver_port=0
--tserver_master_addrs=127.27.157.254:34507
--builtin_ntp_servers=127.27.157.212:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 08:11:06.508607 28508 heartbeater.cc:347] Connected to a master server at 127.27.157.254:34507
I20240503 08:11:06.508955 28508 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:06.510090 28508 heartbeater.cc:511] Master 127.27.157.254:34507 requested a full tablet report, sending...
I20240503 08:11:06.512503 28322 ts_manager.cc:194] Registered new tserver with Master: a7cb0e6105c149db99971899ca8272e2 (127.27.157.193:37333)
I20240503 08:11:06.514197 28322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.193:44817
W20240503 08:11:06.844065 28512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:06.844405 28512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:06.844554 28512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:06.852475 28512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:06.852730 28512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.194
I20240503 08:11:06.862139 28512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.157.194
--webserver_port=0
--tserver_master_addrs=127.27.157.254:34507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.194
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:06.863807 28512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:06.865636 28512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:06.878935 28520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.879575 28517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:06.880077 28518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:06.881186 28512 server_base.cc:1034] running on GCE node
I20240503 08:11:06.881950 28512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:06.882983 28512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:06.884231 28512 hybrid_clock.cc:648] HybridClock initialized: now 1714723866884126 us; error 72 us; skew 500 ppm
I20240503 08:11:06.884617 28512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:06.887099 28512 webserver.cc:460] Webserver started at http://127.27.157.194:41313/ using document root <none> and password file <none>
I20240503 08:11:06.887921 28512 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:06.888048 28512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:06.888374 28512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:06.891167 28512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f020469231b240fabded9a0c25c643c2"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-dj8w"
I20240503 08:11:06.892016 28512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f020469231b240fabded9a0c25c643c2"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-dj8w"
I20240503 08:11:06.897650 28512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:06.901511 28526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:06.903028 28512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:06.903223 28512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f020469231b240fabded9a0c25c643c2"
format_stamp: "Formatted at 2024-05-03 08:11:06 on dist-test-slave-dj8w"
I20240503 08:11:06.903512 28512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:06.942008 28512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:06.942903 28512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:06.943189 28512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:06.944236 28512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:06.945801 28512 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:06.945874 28512 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:06.945991 28512 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:06.946041 28512 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:06.979418 28512 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.194:42551
I20240503 08:11:06.979521 28638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.194:42551 every 8 connection(s)
I20240503 08:11:06.980813 28512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:06.988258 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 28512
I20240503 08:11:06.988484 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:06.992192 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.195:0
--local_ip_for_outbound_sockets=127.27.157.195
--webserver_interface=127.27.157.195
--webserver_port=0
--tserver_master_addrs=127.27.157.254:34507
--builtin_ntp_servers=127.27.157.212:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 08:11:06.994853 28639 heartbeater.cc:347] Connected to a master server at 127.27.157.254:34507
I20240503 08:11:06.995216 28639 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:06.996395 28639 heartbeater.cc:511] Master 127.27.157.254:34507 requested a full tablet report, sending...
I20240503 08:11:06.997987 28322 ts_manager.cc:194] Registered new tserver with Master: f020469231b240fabded9a0c25c643c2 (127.27.157.194:42551)
I20240503 08:11:06.998792 28322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.194:44723
W20240503 08:11:07.338130 28642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:07.338479 28642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:07.338635 28642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:07.346482 28642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:07.346743 28642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.195
I20240503 08:11:07.356201 28642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.157.195
--webserver_port=0
--tserver_master_addrs=127.27.157.254:34507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.195
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:07.357640 28642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:07.359891 28642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:07.373054 28647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.373855 28648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.374485 28650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.375228 28642 server_base.cc:1034] running on GCE node
I20240503 08:11:07.375929 28642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:07.376927 28642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:07.378154 28642 hybrid_clock.cc:648] HybridClock initialized: now 1714723867378073 us; error 54 us; skew 500 ppm
I20240503 08:11:07.378479 28642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.381129 28642 webserver.cc:460] Webserver started at http://127.27.157.195:40415/ using document root <none> and password file <none>
I20240503 08:11:07.381939 28642 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.382068 28642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.382405 28642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.385229 28642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "db3fe786adc5474ca2f8454a41b72db8"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-dj8w"
I20240503 08:11:07.386049 28642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "db3fe786adc5474ca2f8454a41b72db8"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-dj8w"
I20240503 08:11:07.391654 28642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240503 08:11:07.395357 28656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.396942 28642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:07.397112 28642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "db3fe786adc5474ca2f8454a41b72db8"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-dj8w"
I20240503 08:11:07.397306 28642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.424685 28642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:07.425575 28642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:07.425877 28642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.426908 28642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:07.428536 28642 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:07.428612 28642 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.428736 28642 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:07.428786 28642 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.462699 28642 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.195:42683
I20240503 08:11:07.462739 28768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.195:42683 every 8 connection(s)
I20240503 08:11:07.464102 28642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:07.472057 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 28642
I20240503 08:11:07.472267 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:07.475688 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.196:0
--local_ip_for_outbound_sockets=127.27.157.196
--webserver_interface=127.27.157.196
--webserver_port=0
--tserver_master_addrs=127.27.157.254:34507
--builtin_ntp_servers=127.27.157.212:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 08:11:07.478071 28769 heartbeater.cc:347] Connected to a master server at 127.27.157.254:34507
I20240503 08:11:07.478462 28769 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.479643 28769 heartbeater.cc:511] Master 127.27.157.254:34507 requested a full tablet report, sending...
I20240503 08:11:07.481505 28322 ts_manager.cc:194] Registered new tserver with Master: db3fe786adc5474ca2f8454a41b72db8 (127.27.157.195:42683)
I20240503 08:11:07.482317 28322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.195:42107
I20240503 08:11:07.518072 28508 heartbeater.cc:503] Master 127.27.157.254:34507 was elected leader, sending a full tablet report...
W20240503 08:11:07.826833 28772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:07.827186 28772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:07.827366 28772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:07.835305 28772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:07.835743 28772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.196
I20240503 08:11:07.849241 28772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.157.196
--webserver_port=0
--tserver_master_addrs=127.27.157.254:34507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.196
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:07.850703 28772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:07.852552 28772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:07.865799 28777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.866511 28772 server_base.cc:1034] running on GCE node
W20240503 08:11:07.867070 28780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:07.867985 28778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:07.868762 28772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:07.869849 28772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:07.871124 28772 hybrid_clock.cc:648] HybridClock initialized: now 1714723867871054 us; error 49 us; skew 500 ppm
I20240503 08:11:07.871534 28772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:07.874137 28772 webserver.cc:460] Webserver started at http://127.27.157.196:33343/ using document root <none> and password file <none>
I20240503 08:11:07.874994 28772 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:07.875147 28772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:07.875566 28772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:07.878369 28772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "adce733a18a64959859c1e588d1904f1"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-dj8w"
I20240503 08:11:07.879182 28772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "adce733a18a64959859c1e588d1904f1"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-dj8w"
I20240503 08:11:07.884912 28772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 08:11:07.888664 28786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.890085 28772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:07.890254 28772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "adce733a18a64959859c1e588d1904f1"
format_stamp: "Formatted at 2024-05-03 08:11:07 on dist-test-slave-dj8w"
I20240503 08:11:07.890441 28772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:07.914649 28772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:07.915699 28772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:07.916025 28772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:07.917057 28772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:07.918943 28772 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:07.919034 28772 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.919185 28772 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:07.919253 28772 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:07.958029 28772 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.196:44187
I20240503 08:11:07.958065 28898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.196:44187 every 8 connection(s)
I20240503 08:11:07.959487 28772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:07.966959 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 28772
I20240503 08:11:07.967119 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:07.970757 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.197:0
--local_ip_for_outbound_sockets=127.27.157.197
--webserver_interface=127.27.157.197
--webserver_port=0
--tserver_master_addrs=127.27.157.254:34507
--builtin_ntp_servers=127.27.157.212:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 08:11:07.973815 28899 heartbeater.cc:347] Connected to a master server at 127.27.157.254:34507
I20240503 08:11:07.974201 28899 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:07.975114 28899 heartbeater.cc:511] Master 127.27.157.254:34507 requested a full tablet report, sending...
I20240503 08:11:07.976753 28322 ts_manager.cc:194] Registered new tserver with Master: adce733a18a64959859c1e588d1904f1 (127.27.157.196:44187)
I20240503 08:11:07.977622 28322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.196:42375
I20240503 08:11:08.001461 28639 heartbeater.cc:503] Master 127.27.157.254:34507 was elected leader, sending a full tablet report...
W20240503 08:11:08.314827 28902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:08.315160 28902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:08.315285 28902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:08.323410 28902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:08.323670 28902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.197
I20240503 08:11:08.333518 28902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.157.197
--webserver_port=0
--tserver_master_addrs=127.27.157.254:34507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.197
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:08.334955 28902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:08.336866 28902 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:08.353339 28907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.353950 28910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:08.354679 28908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:08.355446 28902 server_base.cc:1034] running on GCE node
I20240503 08:11:08.356187 28902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:08.357249 28902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:08.358494 28902 hybrid_clock.cc:648] HybridClock initialized: now 1714723868358431 us; error 44 us; skew 500 ppm
I20240503 08:11:08.358839 28902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:08.361542 28902 webserver.cc:460] Webserver started at http://127.27.157.197:33511/ using document root <none> and password file <none>
I20240503 08:11:08.362365 28902 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:08.362506 28902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:08.362816 28902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:08.365685 28902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6d5be928051c43b496a6ee7480dab2fb"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dj8w"
I20240503 08:11:08.366487 28902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6d5be928051c43b496a6ee7480dab2fb"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dj8w"
I20240503 08:11:08.372048 28902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 08:11:08.376488 28916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.378083 28902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:11:08.378288 28902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6d5be928051c43b496a6ee7480dab2fb"
format_stamp: "Formatted at 2024-05-03 08:11:08 on dist-test-slave-dj8w"
I20240503 08:11:08.378496 28902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:08.404338 28902 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:08.405274 28902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:08.405580 28902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:08.406580 28902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:08.408442 28902 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:08.408526 28902 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:08.408664 28902 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:08.408761 28902 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 08:11:08.441954 28902 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.197:33179
I20240503 08:11:08.441985 29028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.197:33179 every 8 connection(s)
I20240503 08:11:08.443492 28902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:08.450979 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 28902
I20240503 08:11:08.451184 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:08.456638 29029 heartbeater.cc:347] Connected to a master server at 127.27.157.254:34507
I20240503 08:11:08.456954 29029 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:08.457779 29029 heartbeater.cc:511] Master 127.27.157.254:34507 requested a full tablet report, sending...
I20240503 08:11:08.459172 28322 ts_manager.cc:194] Registered new tserver with Master: 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197:33179)
I20240503 08:11:08.460014 28322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.197:45741
I20240503 08:11:08.468325 28279 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:08.485718 28769 heartbeater.cc:503] Master 127.27.157.254:34507 was elected leader, sending a full tablet report...
I20240503 08:11:08.490453 28322 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41546:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:11:08.537092 28574 tablet_service.cc:1467] Processing CreateTablet for tablet 2961a38fd9f44762a31c6d52fbcffae0 (DEFAULT_TABLE table=TestTable [id=7a2b8d43ae7149b8ac6b40ebe61fa48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:08.537103 28443 tablet_service.cc:1467] Processing CreateTablet for tablet 2961a38fd9f44762a31c6d52fbcffae0 (DEFAULT_TABLE table=TestTable [id=7a2b8d43ae7149b8ac6b40ebe61fa48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:08.537103 28704 tablet_service.cc:1467] Processing CreateTablet for tablet 2961a38fd9f44762a31c6d52fbcffae0 (DEFAULT_TABLE table=TestTable [id=7a2b8d43ae7149b8ac6b40ebe61fa48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:08.538434 28574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2961a38fd9f44762a31c6d52fbcffae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:08.538563 28704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2961a38fd9f44762a31c6d52fbcffae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:08.538579 28443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2961a38fd9f44762a31c6d52fbcffae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:08.549674 29047 tablet_bootstrap.cc:492] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2: Bootstrap starting.
I20240503 08:11:08.552498 29049 tablet_bootstrap.cc:492] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2: Bootstrap starting.
I20240503 08:11:08.554356 29048 tablet_bootstrap.cc:492] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8: Bootstrap starting.
I20240503 08:11:08.554848 29047 tablet_bootstrap.cc:654] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:08.556221 29049 tablet_bootstrap.cc:654] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:08.556368 29047 log.cc:824] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:08.557473 29049 log.cc:824] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:08.557556 29048 tablet_bootstrap.cc:654] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:08.558727 29048 log.cc:824] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:08.559545 29047 tablet_bootstrap.cc:492] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2: No bootstrap required, opened a new log
I20240503 08:11:08.559854 29047 ts_tablet_manager.cc:1367] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240503 08:11:08.560360 29049 tablet_bootstrap.cc:492] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2: No bootstrap required, opened a new log
I20240503 08:11:08.560674 29049 ts_tablet_manager.cc:1367] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:11:08.561092 29048 tablet_bootstrap.cc:492] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8: No bootstrap required, opened a new log
I20240503 08:11:08.561316 29048 ts_tablet_manager.cc:1367] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 08:11:08.567874 29048 raft_consensus.cc:348] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } }
I20240503 08:11:08.568289 29048 raft_consensus.cc:374] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:08.568413 29048 raft_consensus.cc:729] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3fe786adc5474ca2f8454a41b72db8, State: Initialized, Role: FOLLOWER
I20240503 08:11:08.569149 29048 consensus_queue.cc:260] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } }
I20240503 08:11:08.570235 29047 raft_consensus.cc:348] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } }
I20240503 08:11:08.569797 29049 raft_consensus.cc:348] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } }
I20240503 08:11:08.570689 29049 raft_consensus.cc:374] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:08.570765 29047 raft_consensus.cc:374] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:08.570847 29049 raft_consensus.cc:729] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7cb0e6105c149db99971899ca8272e2, State: Initialized, Role: FOLLOWER
I20240503 08:11:08.570927 29047 raft_consensus.cc:729] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f020469231b240fabded9a0c25c643c2, State: Initialized, Role: FOLLOWER
I20240503 08:11:08.571061 29048 ts_tablet_manager.cc:1398] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240503 08:11:08.571811 29049 consensus_queue.cc:260] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } }
I20240503 08:11:08.571810 29047 consensus_queue.cc:260] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } }
I20240503 08:11:08.573700 29049 ts_tablet_manager.cc:1398] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20240503 08:11:08.576323 29047 ts_tablet_manager.cc:1398] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2: Time spent starting tablet: real 0.016s user 0.010s sys 0.002s
I20240503 08:11:08.577886 29055 raft_consensus.cc:482] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:08.578176 29055 raft_consensus.cc:504] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } }
I20240503 08:11:08.580694 29055 leader_election.cc:288] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db3fe786adc5474ca2f8454a41b72db8 (127.27.157.195:42683), a7cb0e6105c149db99971899ca8272e2 (127.27.157.193:37333)
I20240503 08:11:08.588570 28724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2961a38fd9f44762a31c6d52fbcffae0" candidate_uuid: "f020469231b240fabded9a0c25c643c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3fe786adc5474ca2f8454a41b72db8" is_pre_election: true
I20240503 08:11:08.588847 28463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2961a38fd9f44762a31c6d52fbcffae0" candidate_uuid: "f020469231b240fabded9a0c25c643c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7cb0e6105c149db99971899ca8272e2" is_pre_election: true
I20240503 08:11:08.589099 28724 raft_consensus.cc:2454] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f020469231b240fabded9a0c25c643c2 in term 0.
I20240503 08:11:08.589372 28463 raft_consensus.cc:2454] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f020469231b240fabded9a0c25c643c2 in term 0.
I20240503 08:11:08.589849 28530 leader_election.cc:302] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db3fe786adc5474ca2f8454a41b72db8, f020469231b240fabded9a0c25c643c2; no voters:
I20240503 08:11:08.590371 29055 raft_consensus.cc:2781] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:08.590533 29055 raft_consensus.cc:482] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:08.590662 29055 raft_consensus.cc:3037] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:08.593312 29055 raft_consensus.cc:504] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } }
I20240503 08:11:08.594180 29055 leader_election.cc:288] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [CANDIDATE]: Term 1 election: Requested vote from peers db3fe786adc5474ca2f8454a41b72db8 (127.27.157.195:42683), a7cb0e6105c149db99971899ca8272e2 (127.27.157.193:37333)
I20240503 08:11:08.594765 28724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2961a38fd9f44762a31c6d52fbcffae0" candidate_uuid: "f020469231b240fabded9a0c25c643c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3fe786adc5474ca2f8454a41b72db8"
I20240503 08:11:08.594774 28463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2961a38fd9f44762a31c6d52fbcffae0" candidate_uuid: "f020469231b240fabded9a0c25c643c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7cb0e6105c149db99971899ca8272e2"
I20240503 08:11:08.595023 28724 raft_consensus.cc:3037] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:08.595023 28463 raft_consensus.cc:3037] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:08.598546 28724 raft_consensus.cc:2454] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f020469231b240fabded9a0c25c643c2 in term 1.
I20240503 08:11:08.598546 28463 raft_consensus.cc:2454] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f020469231b240fabded9a0c25c643c2 in term 1.
I20240503 08:11:08.599154 28529 leader_election.cc:302] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7cb0e6105c149db99971899ca8272e2, f020469231b240fabded9a0c25c643c2; no voters:
I20240503 08:11:08.599597 29055 raft_consensus.cc:2781] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:08.600065 29055 raft_consensus.cc:686] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 1 LEADER]: Becoming Leader. State: Replica: f020469231b240fabded9a0c25c643c2, State: Running, Role: LEADER
I20240503 08:11:08.600620 29055 consensus_queue.cc:237] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } }
I20240503 08:11:08.605463 28321 catalog_manager.cc:5484] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f020469231b240fabded9a0c25c643c2 (127.27.157.194). New cstate: current_term: 1 leader_uuid: "f020469231b240fabded9a0c25c643c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:08.621878 28279 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:08.624694 28279 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a7cb0e6105c149db99971899ca8272e2 to finish bootstrapping
I20240503 08:11:08.635047 28279 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f020469231b240fabded9a0c25c643c2 to finish bootstrapping
I20240503 08:11:08.643555 28279 ts_itest-base.cc:233] Waiting for 1 tablets on tserver db3fe786adc5474ca2f8454a41b72db8 to finish bootstrapping
I20240503 08:11:08.652506 28279 ts_itest-base.cc:233] Waiting for 0 tablets on tserver adce733a18a64959859c1e588d1904f1 to finish bootstrapping
I20240503 08:11:08.662601 28279 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6d5be928051c43b496a6ee7480dab2fb to finish bootstrapping
W20240503 08:11:08.716791 28770 tablet.cc:2369] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:08.733801 28640 tablet.cc:2369] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:08.747690 28509 tablet.cc:2369] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:08.980468 28899 heartbeater.cc:503] Master 127.27.157.254:34507 was elected leader, sending a full tablet report...
I20240503 08:11:09.124509 29086 consensus_queue.cc:1035] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:09.143972 29089 consensus_queue.cc:1035] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:09.228760 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:09.237841 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:09.238903 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:09.239542 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:09.267237 28574 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:09.462667 29029 heartbeater.cc:503] Master 127.27.157.254:34507 was elected leader, sending a full tablet report...
I20240503 08:11:09.885519 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:09.885996 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:09.894083 28574 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:09.896824 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:09.904840 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.072510 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.085129 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.088212 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.089555 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.096742 28574 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:10.271526 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.273761 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.276365 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.280411 28574 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:10.289181 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:10.837888 28322 ts_manager.cc:292] Set tserver state for a7cb0e6105c149db99971899ca8272e2 to MAINTENANCE_MODE
I20240503 08:11:12.337937 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.347584 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.348766 28574 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:12.349447 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.362929 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.532785 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.539746 28574 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:12.539862 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.546778 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.546774 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.743428 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.755757 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.757658 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.763597 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:12.777515 28574 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:12.841137 28594 consensus_queue.cc:237] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } }
I20240503 08:11:12.843935 28724 raft_consensus.cc:1261] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Refusing update from remote peer f020469231b240fabded9a0c25c643c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:12.843907 28463 raft_consensus.cc:1261] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 1 FOLLOWER]: Refusing update from remote peer f020469231b240fabded9a0c25c643c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:12.844482 29089 consensus_queue.cc:1035] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:12.844808 29341 consensus_queue.cc:1035] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:11:12.848273 29089 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } }
I20240503 08:11:12.848699 28724 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } }
I20240503 08:11:12.848937 28463 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } }
I20240503 08:11:12.901224 28321 catalog_manager.cc:5484] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "f020469231b240fabded9a0c25c643c2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240503 08:11:12.912720 28594 consensus_queue.cc:237] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: true } }
I20240503 08:11:12.916121 28463 raft_consensus.cc:1261] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 1 FOLLOWER]: Refusing update from remote peer f020469231b240fabded9a0c25c643c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:11:12.916227 28724 raft_consensus.cc:1261] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Refusing update from remote peer f020469231b240fabded9a0c25c643c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:11:12.916733 29089 consensus_queue.cc:1035] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:12.917096 29341 consensus_queue.cc:1035] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:11:12.919890 29341 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: true } } }
I20240503 08:11:12.920455 28463 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: true } } }
I20240503 08:11:12.921999 28724 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: true } } }
W20240503 08:11:12.925712 28528 consensus_peers.cc:485] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 -> Peer 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197:33179): Couldn't send request to peer 6d5be928051c43b496a6ee7480dab2fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2961a38fd9f44762a31c6d52fbcffae0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:13.008805 28307 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2961a38fd9f44762a31c6d52fbcffae0 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:11:13.010501 28320 catalog_manager.cc:5484] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 reported cstate change: config changed from index 2 to 3, NON_VOTER 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "f020469231b240fabded9a0c25c643c2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:13.418706 29367 ts_tablet_manager.cc:902] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb: Initiating tablet copy from peer f020469231b240fabded9a0c25c643c2 (127.27.157.194:42551)
I20240503 08:11:13.420568 29367 tablet_copy_client.cc:308] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:42551
I20240503 08:11:13.427513 28614 tablet_copy_service.cc:140] P f020469231b240fabded9a0c25c643c2: Received BeginTabletCopySession request for tablet 2961a38fd9f44762a31c6d52fbcffae0 from peer 6d5be928051c43b496a6ee7480dab2fb ({username='slave'} at 127.27.157.197:52483)
I20240503 08:11:13.427760 28614 tablet_copy_service.cc:161] P f020469231b240fabded9a0c25c643c2: Beginning new tablet copy session on tablet 2961a38fd9f44762a31c6d52fbcffae0 from peer 6d5be928051c43b496a6ee7480dab2fb at {username='slave'} at 127.27.157.197:52483: session id = 6d5be928051c43b496a6ee7480dab2fb-2961a38fd9f44762a31c6d52fbcffae0
I20240503 08:11:13.430693 28614 tablet_copy_source_session.cc:205] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:13.435312 29367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2961a38fd9f44762a31c6d52fbcffae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:13.445372 29367 tablet_copy_client.cc:783] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:13.445778 29367 tablet_copy_client.cc:647] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:13.448333 29367 tablet_copy_client.cc:523] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:13.506086 29367 tablet_bootstrap.cc:492] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb: Bootstrap starting.
I20240503 08:11:13.513931 29367 log.cc:824] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:13.517923 29367 tablet_bootstrap.cc:492] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:13.518355 29367 tablet_bootstrap.cc:492] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb: Bootstrap complete.
I20240503 08:11:13.518805 29367 ts_tablet_manager.cc:1367] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240503 08:11:13.525206 29367 raft_consensus.cc:348] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: true } }
I20240503 08:11:13.525724 29367 raft_consensus.cc:729] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d5be928051c43b496a6ee7480dab2fb, State: Initialized, Role: LEARNER
I20240503 08:11:13.526273 29367 consensus_queue.cc:260] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: true } }
I20240503 08:11:13.527935 29367 ts_tablet_manager.cc:1398] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb: Time spent starting tablet: real 0.009s user 0.008s sys 0.004s
I20240503 08:11:13.529071 28614 tablet_copy_service.cc:342] P f020469231b240fabded9a0c25c643c2: Request end of tablet copy session 6d5be928051c43b496a6ee7480dab2fb-2961a38fd9f44762a31c6d52fbcffae0 received from {username='slave'} at 127.27.157.197:52483
I20240503 08:11:13.529273 28614 tablet_copy_service.cc:433] P f020469231b240fabded9a0c25c643c2: ending tablet copy session 6d5be928051c43b496a6ee7480dab2fb-2961a38fd9f44762a31c6d52fbcffae0 on tablet 2961a38fd9f44762a31c6d52fbcffae0 with peer 6d5be928051c43b496a6ee7480dab2fb
I20240503 08:11:14.034081 28984 raft_consensus.cc:1203] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240503 08:11:14.507457 29390 raft_consensus.cc:1050] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2: attempting to promote NON_VOTER 6d5be928051c43b496a6ee7480dab2fb to VOTER
I20240503 08:11:14.508380 29390 consensus_queue.cc:237] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } }
I20240503 08:11:14.511556 28463 raft_consensus.cc:1261] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 1 FOLLOWER]: Refusing update from remote peer f020469231b240fabded9a0c25c643c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:14.512138 28984 raft_consensus.cc:1261] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [term 1 LEARNER]: Refusing update from remote peer f020469231b240fabded9a0c25c643c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:14.512239 29344 consensus_queue.cc:1035] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:14.512362 28724 raft_consensus.cc:1261] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Refusing update from remote peer f020469231b240fabded9a0c25c643c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:11:14.512964 29344 consensus_queue.cc:1035] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:14.513320 29404 consensus_queue.cc:1035] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:11:14.516528 29344 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } } }
I20240503 08:11:14.518348 28984 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } } }
I20240503 08:11:14.519892 28724 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } } }
I20240503 08:11:14.609886 28463 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } } }
I20240503 08:11:14.612323 28321 catalog_manager.cc:5484] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 reported cstate change: config changed from index 3 to 4, 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f020469231b240fabded9a0c25c643c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7cb0e6105c149db99971899ca8272e2" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 37333 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:14.621075 28593 consensus_queue.cc:237] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } }
I20240503 08:11:14.623133 28984 raft_consensus.cc:1261] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [term 1 FOLLOWER]: Refusing update from remote peer f020469231b240fabded9a0c25c643c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:14.623162 28724 raft_consensus.cc:1261] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Refusing update from remote peer f020469231b240fabded9a0c25c643c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:14.623760 29390 consensus_queue.cc:1035] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:14.624195 29391 consensus_queue.cc:1035] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:14.626997 29404 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P f020469231b240fabded9a0c25c643c2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7cb0e6105c149db99971899ca8272e2 (127.27.157.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } } }
I20240503 08:11:14.627748 28984 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7cb0e6105c149db99971899ca8272e2 (127.27.157.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } } }
I20240503 08:11:14.627748 28724 raft_consensus.cc:2932] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7cb0e6105c149db99971899ca8272e2 (127.27.157.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } } }
I20240503 08:11:14.706794 28307 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2961a38fd9f44762a31c6d52fbcffae0 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:11:14.709393 28321 catalog_manager.cc:5484] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb reported cstate change: config changed from index 4 to 5, VOTER a7cb0e6105c149db99971899ca8272e2 (127.27.157.193) evicted. New cstate: current_term: 1 leader_uuid: "f020469231b240fabded9a0c25c643c2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } } }
I20240503 08:11:14.714500 28443 tablet_service.cc:1514] Processing DeleteTablet for tablet 2961a38fd9f44762a31c6d52fbcffae0 with delete_type TABLET_DATA_TOMBSTONED (TS a7cb0e6105c149db99971899ca8272e2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39734
I20240503 08:11:14.715634 29420 tablet_replica.cc:330] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2: stopping tablet replica
I20240503 08:11:14.716074 29420 raft_consensus.cc:2229] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:14.716704 29420 raft_consensus.cc:2258] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:14.718645 29420 ts_tablet_manager.cc:1875] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:14.909687 29420 ts_tablet_manager.cc:1888] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:11:14.909960 29420 log.cc:1196] T 2961a38fd9f44762a31c6d52fbcffae0 P a7cb0e6105c149db99971899ca8272e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/2961a38fd9f44762a31c6d52fbcffae0
I20240503 08:11:14.911111 28308 catalog_manager.cc:4830] TS a7cb0e6105c149db99971899ca8272e2 (127.27.157.193:37333): tablet 2961a38fd9f44762a31c6d52fbcffae0 (table TestTable [id=7a2b8d43ae7149b8ac6b40ebe61fa48e]) successfully deleted
I20240503 08:11:15.136982 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.138027 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.153812 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.154253 28574 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:15.154389 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.318056 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.321115 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.322834 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.330399 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.332370 28574 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:15.495782 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.496569 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.498405 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.501134 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:15.501345 28574 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:15.686275 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 28512
I20240503 08:11:15.687963 28633 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:15.807368 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 28512
W20240503 08:11:15.827301 28917 connection.cc:535] server connection from 127.27.157.194:34481 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:11:15.827740 28306 connection.cc:535] server connection from 127.27.157.194:44723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:16.329350 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.329661 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.329815 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.339911 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.475695 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.476864 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.478479 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.483651 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.669677 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.673806 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.682564 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.684840 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.839555 28443 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.840721 28964 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.846999 28704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.848194 28834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:16.986245 29640 raft_consensus.cc:482] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f020469231b240fabded9a0c25c643c2)
I20240503 08:11:16.986500 29640 raft_consensus.cc:504] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } }
I20240503 08:11:16.988423 29640 leader_election.cc:288] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f020469231b240fabded9a0c25c643c2 (127.27.157.194:42551), 6d5be928051c43b496a6ee7480dab2fb (127.27.157.197:33179)
W20240503 08:11:16.990655 28658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.157.194:42551: connect: Connection refused (error 111)
I20240503 08:11:17.003639 28984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2961a38fd9f44762a31c6d52fbcffae0" candidate_uuid: "db3fe786adc5474ca2f8454a41b72db8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6d5be928051c43b496a6ee7480dab2fb" is_pre_election: true
W20240503 08:11:17.003953 28658 leader_election.cc:334] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f020469231b240fabded9a0c25c643c2 (127.27.157.194:42551): Network error: Client connection negotiation failed: client connection to 127.27.157.194:42551: connect: Connection refused (error 111)
I20240503 08:11:17.004611 28658 leader_election.cc:302] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db3fe786adc5474ca2f8454a41b72db8; no voters: 6d5be928051c43b496a6ee7480dab2fb, f020469231b240fabded9a0c25c643c2
I20240503 08:11:17.004978 29640 raft_consensus.cc:2726] T 2961a38fd9f44762a31c6d52fbcffae0 P db3fe786adc5474ca2f8454a41b72db8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:17.009544 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 28381
I20240503 08:11:17.011147 28503 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:17.129761 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 28381
I20240503 08:11:17.150979 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 28642
I20240503 08:11:17.152280 28763 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:17.274613 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 28642
I20240503 08:11:17.296475 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 28772
I20240503 08:11:17.297763 28893 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:17.308948 29647 raft_consensus.cc:482] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f020469231b240fabded9a0c25c643c2)
I20240503 08:11:17.309238 29647 raft_consensus.cc:504] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f020469231b240fabded9a0c25c643c2" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 42551 } } peers { permanent_uuid: "db3fe786adc5474ca2f8454a41b72db8" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42683 } } peers { permanent_uuid: "6d5be928051c43b496a6ee7480dab2fb" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 33179 } attrs { promote: false } }
I20240503 08:11:17.310981 29647 leader_election.cc:288] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f020469231b240fabded9a0c25c643c2 (127.27.157.194:42551), db3fe786adc5474ca2f8454a41b72db8 (127.27.157.195:42683)
W20240503 08:11:17.312233 28918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.157.194:42551: connect: Connection refused (error 111)
W20240503 08:11:17.314082 28918 leader_election.cc:334] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f020469231b240fabded9a0c25c643c2 (127.27.157.194:42551): Network error: Client connection negotiation failed: client connection to 127.27.157.194:42551: connect: Connection refused (error 111)
W20240503 08:11:17.314370 28920 leader_election.cc:334] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db3fe786adc5474ca2f8454a41b72db8 (127.27.157.195:42683): Network error: Client connection negotiation failed: client connection to 127.27.157.195:42683: connect: Connection refused (error 111)
I20240503 08:11:17.314515 28920 leader_election.cc:302] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d5be928051c43b496a6ee7480dab2fb; no voters: db3fe786adc5474ca2f8454a41b72db8, f020469231b240fabded9a0c25c643c2
I20240503 08:11:17.314875 29647 raft_consensus.cc:2726] T 2961a38fd9f44762a31c6d52fbcffae0 P 6d5be928051c43b496a6ee7480dab2fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:11:17.414861 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 28772
I20240503 08:11:17.435572 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 28902
I20240503 08:11:17.436807 29024 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:17.556941 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 28902
I20240503 08:11:17.579807 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 28291
I20240503 08:11:17.581629 28352 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:17.726543 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 28291
2024-05-03T08:11:17Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (12322 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (12322 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240503 08:11:17.774230 28279 test_util.cc:268] Using random seed: 1134519142
I20240503 08:11:17.776293 28279 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:17.776388 28279 ts_itest-base.cc:113] --------------
I20240503 08:11:17.776424 28279 ts_itest-base.cc:114] 5 tablet servers
I20240503 08:11:17.776494 28279 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:11:17.776546 28279 ts_itest-base.cc:116] --------------
2024-05-03T08:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:17Z Disabled control of system clock
I20240503 08:11:17.802646 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.157.254:42457
--webserver_interface=127.27.157.254
--webserver_port=0
--builtin_ntp_servers=127.27.157.212:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.157.254:42457
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 08:11:18.145339 29660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:18.145709 29660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:18.145824 29660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:18.154107 29660 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:18.154273 29660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:18.154325 29660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:18.154453 29660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:18.154525 29660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:18.164834 29660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.157.254:42457
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.157.254:42457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.157.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:18.166321 29660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:18.168499 29660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:18.181890 29666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:18.182478 29668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:18.183614 29665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:18.184645 29660 server_base.cc:1034] running on GCE node
I20240503 08:11:18.185634 29660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:18.187297 29660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:18.188632 29660 hybrid_clock.cc:648] HybridClock initialized: now 1714723878188586 us; error 41 us; skew 500 ppm
I20240503 08:11:18.189085 29660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:18.191818 29660 webserver.cc:460] Webserver started at http://127.27.157.254:42011/ using document root <none> and password file <none>
I20240503 08:11:18.192660 29660 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:18.192785 29660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:18.193156 29660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:18.196317 29660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0c2a80e41be64cacbba4c32c7d699ebc"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-dj8w"
I20240503 08:11:18.197335 29660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0c2a80e41be64cacbba4c32c7d699ebc"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-dj8w"
I20240503 08:11:18.203671 29660 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240503 08:11:18.207772 29674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.209574 29660 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:11:18.209805 29660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0c2a80e41be64cacbba4c32c7d699ebc"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-dj8w"
I20240503 08:11:18.210045 29660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:18.233315 29660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:18.234261 29660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:18.234557 29660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:18.254299 29660 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.254:42457
I20240503 08:11:18.254343 29725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.254:42457 every 8 connection(s)
I20240503 08:11:18.255831 29660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:18.260211 29726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:18.262961 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 29660
I20240503 08:11:18.263208 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:18.272694 29726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc: Bootstrap starting.
I20240503 08:11:18.276281 29726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:18.277707 29726 log.cc:824] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:18.280754 29726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc: No bootstrap required, opened a new log
I20240503 08:11:18.287263 29726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2a80e41be64cacbba4c32c7d699ebc" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 42457 } }
I20240503 08:11:18.287691 29726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:18.287781 29726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2a80e41be64cacbba4c32c7d699ebc, State: Initialized, Role: FOLLOWER
I20240503 08:11:18.288389 29726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2a80e41be64cacbba4c32c7d699ebc" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 42457 } }
I20240503 08:11:18.288592 29726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:18.288703 29726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:18.288856 29726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:18.291198 29726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2a80e41be64cacbba4c32c7d699ebc" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 42457 } }
I20240503 08:11:18.291785 29726 leader_election.cc:302] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c2a80e41be64cacbba4c32c7d699ebc; no voters:
I20240503 08:11:18.292373 29726 leader_election.cc:288] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:18.292565 29731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:18.293205 29731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [term 1 LEADER]: Becoming Leader. State: Replica: 0c2a80e41be64cacbba4c32c7d699ebc, State: Running, Role: LEADER
I20240503 08:11:18.293818 29731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2a80e41be64cacbba4c32c7d699ebc" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 42457 } }
I20240503 08:11:18.294494 29726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:18.297431 29732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c2a80e41be64cacbba4c32c7d699ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2a80e41be64cacbba4c32c7d699ebc" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 42457 } } }
I20240503 08:11:18.297169 29733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c2a80e41be64cacbba4c32c7d699ebc. Latest consensus state: current_term: 1 leader_uuid: "0c2a80e41be64cacbba4c32c7d699ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2a80e41be64cacbba4c32c7d699ebc" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 42457 } } }
I20240503 08:11:18.297920 29732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:18.298666 29733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:18.300493 29737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:18.306982 29737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:18.316900 29737 catalog_manager.cc:1347] Generated new cluster ID: 1363f5b6a5944e16bf2c07051f5a2e11
I20240503 08:11:18.317046 29737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:18.329478 29737 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:18.331557 29737 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:18.350939 29737 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0c2a80e41be64cacbba4c32c7d699ebc: Generated new TSK 0
I20240503 08:11:18.352146 29737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:18.369103 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.193:0
--local_ip_for_outbound_sockets=127.27.157.193
--webserver_interface=127.27.157.193
--webserver_port=0
--tserver_master_addrs=127.27.157.254:42457
--builtin_ntp_servers=127.27.157.212:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:18.727291 29750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:18.727694 29750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:18.727852 29750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:18.735677 29750 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:18.735858 29750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:18.736022 29750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.193
I20240503 08:11:18.745709 29750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.157.193
--webserver_port=0
--tserver_master_addrs=127.27.157.254:42457
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.193
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:18.747159 29750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:18.749082 29750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:18.762382 29756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:18.763233 29755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:18.764102 29758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:18.764946 29750 server_base.cc:1034] running on GCE node
I20240503 08:11:18.765626 29750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:18.766695 29750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:18.767925 29750 hybrid_clock.cc:648] HybridClock initialized: now 1714723878767851 us; error 57 us; skew 500 ppm
I20240503 08:11:18.768280 29750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:18.770875 29750 webserver.cc:460] Webserver started at http://127.27.157.193:33759/ using document root <none> and password file <none>
I20240503 08:11:18.771749 29750 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:18.771893 29750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:18.772246 29750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:18.775178 29750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2a33001abc6a4e82b6caf95841b07bf0"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-dj8w"
I20240503 08:11:18.776082 29750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2a33001abc6a4e82b6caf95841b07bf0"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-dj8w"
I20240503 08:11:18.782338 29750 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20240503 08:11:18.786609 29764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.788542 29750 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:18.788762 29750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2a33001abc6a4e82b6caf95841b07bf0"
format_stamp: "Formatted at 2024-05-03 08:11:18 on dist-test-slave-dj8w"
I20240503 08:11:18.788993 29750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:18.816321 29750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:18.817305 29750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:18.817596 29750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:18.818657 29750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:18.820386 29750 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:18.820495 29750 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.820623 29750 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:18.820698 29750 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:18.855895 29750 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.193:39577
I20240503 08:11:18.855960 29876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.193:39577 every 8 connection(s)
I20240503 08:11:18.857797 29750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:18.859814 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 29750
I20240503 08:11:18.860018 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:18.864784 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.194:0
--local_ip_for_outbound_sockets=127.27.157.194
--webserver_interface=127.27.157.194
--webserver_port=0
--tserver_master_addrs=127.27.157.254:42457
--builtin_ntp_servers=127.27.157.212:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:18.873581 29877 heartbeater.cc:347] Connected to a master server at 127.27.157.254:42457
I20240503 08:11:18.874022 29877 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:18.875093 29877 heartbeater.cc:511] Master 127.27.157.254:42457 requested a full tablet report, sending...
I20240503 08:11:18.877625 29691 ts_manager.cc:194] Registered new tserver with Master: 2a33001abc6a4e82b6caf95841b07bf0 (127.27.157.193:39577)
I20240503 08:11:18.879249 29691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.193:35591
W20240503 08:11:19.216486 29881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:19.216826 29881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:19.216955 29881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:19.225013 29881 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:19.225167 29881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:19.225368 29881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.194
I20240503 08:11:19.235020 29881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.157.194
--webserver_port=0
--tserver_master_addrs=127.27.157.254:42457
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.194
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:19.236479 29881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:19.238286 29881 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:19.251871 29889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:19.252547 29887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:19.253495 29886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:19.254285 29881 server_base.cc:1034] running on GCE node
I20240503 08:11:19.255007 29881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:19.256170 29881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:19.257409 29881 hybrid_clock.cc:648] HybridClock initialized: now 1714723879257369 us; error 48 us; skew 500 ppm
I20240503 08:11:19.257756 29881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:19.260479 29881 webserver.cc:460] Webserver started at http://127.27.157.194:40615/ using document root <none> and password file <none>
I20240503 08:11:19.261332 29881 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:19.261456 29881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:19.261818 29881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:19.265115 29881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5a2a2bcf493e4b88bb329e7411c70940"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-dj8w"
I20240503 08:11:19.266358 29881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5a2a2bcf493e4b88bb329e7411c70940"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-dj8w"
I20240503 08:11:19.272485 29881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20240503 08:11:19.276620 29895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.278121 29881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:11:19.278304 29881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5a2a2bcf493e4b88bb329e7411c70940"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-dj8w"
I20240503 08:11:19.278529 29881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:19.304356 29881 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:19.305289 29881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:19.305580 29881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:19.306653 29881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:19.308344 29881 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:19.308437 29881 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.308559 29881 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:19.308629 29881 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.344599 29881 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.194:40819
I20240503 08:11:19.344645 30007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.194:40819 every 8 connection(s)
I20240503 08:11:19.346091 29881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:19.355373 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 29881
I20240503 08:11:19.355602 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:19.359315 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.195:0
--local_ip_for_outbound_sockets=127.27.157.195
--webserver_interface=127.27.157.195
--webserver_port=0
--tserver_master_addrs=127.27.157.254:42457
--builtin_ntp_servers=127.27.157.212:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:19.360265 30008 heartbeater.cc:347] Connected to a master server at 127.27.157.254:42457
I20240503 08:11:19.360664 30008 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:19.361836 30008 heartbeater.cc:511] Master 127.27.157.254:42457 requested a full tablet report, sending...
I20240503 08:11:19.363726 29691 ts_manager.cc:194] Registered new tserver with Master: 5a2a2bcf493e4b88bb329e7411c70940 (127.27.157.194:40819)
I20240503 08:11:19.364574 29691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.194:59421
W20240503 08:11:19.699795 30011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:19.700150 30011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:19.700263 30011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:19.708206 30011 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:19.708356 30011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:19.708477 30011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.195
I20240503 08:11:19.717978 30011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.157.195
--webserver_port=0
--tserver_master_addrs=127.27.157.254:42457
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.195
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:19.719420 30011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:19.721261 30011 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:19.734552 30016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:19.735213 30017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:19.736279 30019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:19.737286 30011 server_base.cc:1034] running on GCE node
I20240503 08:11:19.737977 30011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:19.739054 30011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:19.740304 30011 hybrid_clock.cc:648] HybridClock initialized: now 1714723879740225 us; error 69 us; skew 500 ppm
I20240503 08:11:19.740659 30011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:19.743155 30011 webserver.cc:460] Webserver started at http://127.27.157.195:44907/ using document root <none> and password file <none>
I20240503 08:11:19.744060 30011 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:19.744194 30011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:19.744553 30011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:19.747489 30011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ee2c63f4e028410c99f5132d75489b53"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-dj8w"
I20240503 08:11:19.748387 30011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ee2c63f4e028410c99f5132d75489b53"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-dj8w"
I20240503 08:11:19.754109 30011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:19.758019 30025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.759765 30011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 08:11:19.759970 30011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ee2c63f4e028410c99f5132d75489b53"
format_stamp: "Formatted at 2024-05-03 08:11:19 on dist-test-slave-dj8w"
I20240503 08:11:19.760196 30011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:19.789949 30011 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:19.790884 30011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:19.791164 30011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:19.792207 30011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:19.793756 30011 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:19.793853 30011 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.793974 30011 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:19.794045 30011 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:19.826049 30011 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.195:44917
I20240503 08:11:19.826124 30137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.195:44917 every 8 connection(s)
I20240503 08:11:19.827498 30011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:19.830595 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 30011
I20240503 08:11:19.830826 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:19.834658 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.196:0
--local_ip_for_outbound_sockets=127.27.157.196
--webserver_interface=127.27.157.196
--webserver_port=0
--tserver_master_addrs=127.27.157.254:42457
--builtin_ntp_servers=127.27.157.212:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:19.843715 30138 heartbeater.cc:347] Connected to a master server at 127.27.157.254:42457
I20240503 08:11:19.844070 30138 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:19.844955 30138 heartbeater.cc:511] Master 127.27.157.254:42457 requested a full tablet report, sending...
I20240503 08:11:19.846601 29691 ts_manager.cc:194] Registered new tserver with Master: ee2c63f4e028410c99f5132d75489b53 (127.27.157.195:44917)
I20240503 08:11:19.847918 29691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.195:58441
I20240503 08:11:19.882779 29877 heartbeater.cc:503] Master 127.27.157.254:42457 was elected leader, sending a full tablet report...
W20240503 08:11:20.178943 30141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:20.179263 30141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:20.179386 30141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:20.187268 30141 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:20.187470 30141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:20.187666 30141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.196
I20240503 08:11:20.197485 30141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.157.196
--webserver_port=0
--tserver_master_addrs=127.27.157.254:42457
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.196
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:20.198863 30141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:20.200778 30141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:20.213689 30146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:20.214241 30147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:20.215276 30149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:20.216229 30141 server_base.cc:1034] running on GCE node
I20240503 08:11:20.216841 30141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:20.217867 30141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:20.219074 30141 hybrid_clock.cc:648] HybridClock initialized: now 1714723880219032 us; error 34 us; skew 500 ppm
I20240503 08:11:20.219470 30141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:20.221957 30141 webserver.cc:460] Webserver started at http://127.27.157.196:41985/ using document root <none> and password file <none>
I20240503 08:11:20.222765 30141 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:20.222887 30141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:20.223249 30141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:20.226154 30141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0746f8bb491a45df8d964c43c37dd59c"
format_stamp: "Formatted at 2024-05-03 08:11:20 on dist-test-slave-dj8w"
I20240503 08:11:20.226953 30141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0746f8bb491a45df8d964c43c37dd59c"
format_stamp: "Formatted at 2024-05-03 08:11:20 on dist-test-slave-dj8w"
I20240503 08:11:20.232735 30141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 08:11:20.236660 30155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:20.238243 30141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:20.238448 30141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0746f8bb491a45df8d964c43c37dd59c"
format_stamp: "Formatted at 2024-05-03 08:11:20 on dist-test-slave-dj8w"
I20240503 08:11:20.238703 30141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:20.254879 30141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:20.255900 30141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:20.256189 30141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:20.257306 30141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:20.259054 30141 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:20.259166 30141 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:20.259294 30141 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:20.259390 30141 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240503 08:11:20.293262 30141 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.196:38913
I20240503 08:11:20.293345 30267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.196:38913 every 8 connection(s)
I20240503 08:11:20.294734 30141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:20.295073 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 30141
I20240503 08:11:20.295282 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:20.299074 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.197:0
--local_ip_for_outbound_sockets=127.27.157.197
--webserver_interface=127.27.157.197
--webserver_port=0
--tserver_master_addrs=127.27.157.254:42457
--builtin_ntp_servers=127.27.157.212:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:20.309480 30268 heartbeater.cc:347] Connected to a master server at 127.27.157.254:42457
I20240503 08:11:20.309818 30268 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:20.310760 30268 heartbeater.cc:511] Master 127.27.157.254:42457 requested a full tablet report, sending...
I20240503 08:11:20.312502 29691 ts_manager.cc:194] Registered new tserver with Master: 0746f8bb491a45df8d964c43c37dd59c (127.27.157.196:38913)
I20240503 08:11:20.313787 29691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.196:37553
I20240503 08:11:20.367940 30008 heartbeater.cc:503] Master 127.27.157.254:42457 was elected leader, sending a full tablet report...
W20240503 08:11:20.656502 30271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:20.656960 30271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:20.657174 30271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:20.665921 30271 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:20.666107 30271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:20.666272 30271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.197
I20240503 08:11:20.676470 30271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.157.197
--webserver_port=0
--tserver_master_addrs=127.27.157.254:42457
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.197
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:20.677922 30271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:20.679853 30271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:20.693589 30276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:20.694187 30277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:20.694931 30279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:20.696033 30271 server_base.cc:1034] running on GCE node
I20240503 08:11:20.696684 30271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:20.697764 30271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:20.699002 30271 hybrid_clock.cc:648] HybridClock initialized: now 1714723880698938 us; error 56 us; skew 500 ppm
I20240503 08:11:20.699391 30271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:20.703794 30271 webserver.cc:460] Webserver started at http://127.27.157.197:44593/ using document root <none> and password file <none>
I20240503 08:11:20.704663 30271 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:20.704785 30271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:20.705158 30271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:20.708253 30271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f415ea6340664cc5ae70fa0db5095545"
format_stamp: "Formatted at 2024-05-03 08:11:20 on dist-test-slave-dj8w"
I20240503 08:11:20.709066 30271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f415ea6340664cc5ae70fa0db5095545"
format_stamp: "Formatted at 2024-05-03 08:11:20 on dist-test-slave-dj8w"
I20240503 08:11:20.715086 30271 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20240503 08:11:20.719617 30285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:20.721346 30271 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:11:20.721532 30271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f415ea6340664cc5ae70fa0db5095545"
format_stamp: "Formatted at 2024-05-03 08:11:20 on dist-test-slave-dj8w"
I20240503 08:11:20.721748 30271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:20.743872 30271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:20.744875 30271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:20.745173 30271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:20.746238 30271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:20.748024 30271 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:20.748127 30271 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:20.748227 30271 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:20.748306 30271 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:20.786137 30271 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.197:46799
I20240503 08:11:20.786149 30397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.197:46799 every 8 connection(s)
I20240503 08:11:20.787997 30271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:20.789949 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 30271
I20240503 08:11:20.790092 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:20.801906 30398 heartbeater.cc:347] Connected to a master server at 127.27.157.254:42457
I20240503 08:11:20.802248 30398 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:20.803169 30398 heartbeater.cc:511] Master 127.27.157.254:42457 requested a full tablet report, sending...
I20240503 08:11:20.804811 29691 ts_manager.cc:194] Registered new tserver with Master: f415ea6340664cc5ae70fa0db5095545 (127.27.157.197:46799)
I20240503 08:11:20.805848 29691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.197:40571
I20240503 08:11:20.806315 28279 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:20.824883 29691 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49958:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:11:20.850783 30138 heartbeater.cc:503] Master 127.27.157.254:42457 was elected leader, sending a full tablet report...
I20240503 08:11:20.860594 29812 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6f32fcd7b44255a18756ce7d2d44f2 (DEFAULT_TABLE table=TestTable [id=0c894dd1297346b089f88d2f6f056518]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.860594 30333 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6f32fcd7b44255a18756ce7d2d44f2 (DEFAULT_TABLE table=TestTable [id=0c894dd1297346b089f88d2f6f056518]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.862270 30333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6f32fcd7b44255a18756ce7d2d44f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.862252 29812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6f32fcd7b44255a18756ce7d2d44f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.862447 30203 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6f32fcd7b44255a18756ce7d2d44f2 (DEFAULT_TABLE table=TestTable [id=0c894dd1297346b089f88d2f6f056518]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:20.863834 30203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6f32fcd7b44255a18756ce7d2d44f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:20.875640 30416 tablet_bootstrap.cc:492] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c: Bootstrap starting.
I20240503 08:11:20.877840 30417 tablet_bootstrap.cc:492] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545: Bootstrap starting.
I20240503 08:11:20.880048 30418 tablet_bootstrap.cc:492] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0: Bootstrap starting.
I20240503 08:11:20.881728 30417 tablet_bootstrap.cc:654] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.881830 30416 tablet_bootstrap.cc:654] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.883213 30418 tablet_bootstrap.cc:654] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:20.883195 30417 log.cc:824] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:20.883195 30416 log.cc:824] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:20.884547 30418 log.cc:824] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:20.886567 30417 tablet_bootstrap.cc:492] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545: No bootstrap required, opened a new log
I20240503 08:11:20.886906 30417 ts_tablet_manager.cc:1367] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 08:11:20.887835 30418 tablet_bootstrap.cc:492] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0: No bootstrap required, opened a new log
I20240503 08:11:20.888226 30418 ts_tablet_manager.cc:1367] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20240503 08:11:20.891230 30416 tablet_bootstrap.cc:492] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c: No bootstrap required, opened a new log
I20240503 08:11:20.891615 30416 ts_tablet_manager.cc:1367] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20240503 08:11:20.896320 30418 raft_consensus.cc:348] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } }
I20240503 08:11:20.896423 30417 raft_consensus.cc:348] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } }
I20240503 08:11:20.896760 30418 raft_consensus.cc:374] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.896869 30418 raft_consensus.cc:729] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a33001abc6a4e82b6caf95841b07bf0, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.896903 30417 raft_consensus.cc:374] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.897006 30417 raft_consensus.cc:729] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f415ea6340664cc5ae70fa0db5095545, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.897614 30418 consensus_queue.cc:260] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } }
I20240503 08:11:20.897799 30417 consensus_queue.cc:260] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } }
I20240503 08:11:20.899803 30418 ts_tablet_manager.cc:1398] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20240503 08:11:20.902158 30416 raft_consensus.cc:348] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } }
I20240503 08:11:20.902588 30416 raft_consensus.cc:374] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:20.902710 30416 raft_consensus.cc:729] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0746f8bb491a45df8d964c43c37dd59c, State: Initialized, Role: FOLLOWER
I20240503 08:11:20.903476 30416 consensus_queue.cc:260] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } }
I20240503 08:11:20.905027 30268 heartbeater.cc:503] Master 127.27.157.254:42457 was elected leader, sending a full tablet report...
I20240503 08:11:20.905572 30416 ts_tablet_manager.cc:1398] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20240503 08:11:20.906598 30398 heartbeater.cc:503] Master 127.27.157.254:42457 was elected leader, sending a full tablet report...
I20240503 08:11:20.907281 30417 ts_tablet_manager.cc:1398] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545: Time spent starting tablet: real 0.020s user 0.017s sys 0.004s
I20240503 08:11:20.918946 30422 raft_consensus.cc:482] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:20.919224 30422 raft_consensus.cc:504] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } }
I20240503 08:11:20.921144 30422 leader_election.cc:288] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0746f8bb491a45df8d964c43c37dd59c (127.27.157.196:38913), f415ea6340664cc5ae70fa0db5095545 (127.27.157.197:46799)
I20240503 08:11:20.930541 30223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6f32fcd7b44255a18756ce7d2d44f2" candidate_uuid: "2a33001abc6a4e82b6caf95841b07bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0746f8bb491a45df8d964c43c37dd59c" is_pre_election: true
I20240503 08:11:20.930532 30353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6f32fcd7b44255a18756ce7d2d44f2" candidate_uuid: "2a33001abc6a4e82b6caf95841b07bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f415ea6340664cc5ae70fa0db5095545" is_pre_election: true
I20240503 08:11:20.931087 30223 raft_consensus.cc:2454] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a33001abc6a4e82b6caf95841b07bf0 in term 0.
I20240503 08:11:20.931087 30353 raft_consensus.cc:2454] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a33001abc6a4e82b6caf95841b07bf0 in term 0.
I20240503 08:11:20.931293 30423 raft_consensus.cc:482] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:20.931514 30423 raft_consensus.cc:504] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } }
I20240503 08:11:20.931977 29765 leader_election.cc:302] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0746f8bb491a45df8d964c43c37dd59c, 2a33001abc6a4e82b6caf95841b07bf0; no voters:
I20240503 08:11:20.932461 30422 raft_consensus.cc:2781] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:20.932644 30422 raft_consensus.cc:482] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:20.932776 30422 raft_consensus.cc:3037] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.933652 30423 leader_election.cc:288] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a33001abc6a4e82b6caf95841b07bf0 (127.27.157.193:39577), 0746f8bb491a45df8d964c43c37dd59c (127.27.157.196:38913)
I20240503 08:11:20.942270 30223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6f32fcd7b44255a18756ce7d2d44f2" candidate_uuid: "f415ea6340664cc5ae70fa0db5095545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0746f8bb491a45df8d964c43c37dd59c" is_pre_election: true
I20240503 08:11:20.942579 30223 raft_consensus.cc:2454] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f415ea6340664cc5ae70fa0db5095545 in term 0.
I20240503 08:11:20.942564 29832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6f32fcd7b44255a18756ce7d2d44f2" candidate_uuid: "f415ea6340664cc5ae70fa0db5095545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a33001abc6a4e82b6caf95841b07bf0" is_pre_election: true
I20240503 08:11:20.943205 30286 leader_election.cc:302] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0746f8bb491a45df8d964c43c37dd59c, f415ea6340664cc5ae70fa0db5095545; no voters:
I20240503 08:11:20.943662 30423 raft_consensus.cc:2781] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:20.943883 30423 raft_consensus.cc:482] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:20.944016 30423 raft_consensus.cc:3037] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.988932 30422 raft_consensus.cc:504] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } }
I20240503 08:11:20.988932 30423 raft_consensus.cc:504] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } }
I20240503 08:11:20.989683 29832 raft_consensus.cc:2379] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f415ea6340664cc5ae70fa0db5095545 in current term 1: Already voted for candidate 2a33001abc6a4e82b6caf95841b07bf0 in this term.
I20240503 08:11:20.989832 30422 leader_election.cc:288] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [CANDIDATE]: Term 1 election: Requested vote from peers 0746f8bb491a45df8d964c43c37dd59c (127.27.157.196:38913), f415ea6340664cc5ae70fa0db5095545 (127.27.157.197:46799)
I20240503 08:11:20.989833 30423 leader_election.cc:288] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [CANDIDATE]: Term 1 election: Requested vote from peers 2a33001abc6a4e82b6caf95841b07bf0 (127.27.157.193:39577), 0746f8bb491a45df8d964c43c37dd59c (127.27.157.196:38913)
I20240503 08:11:20.990666 30223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6f32fcd7b44255a18756ce7d2d44f2" candidate_uuid: "f415ea6340664cc5ae70fa0db5095545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0746f8bb491a45df8d964c43c37dd59c"
I20240503 08:11:20.990932 30223 raft_consensus.cc:3037] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:20.990690 30222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6f32fcd7b44255a18756ce7d2d44f2" candidate_uuid: "2a33001abc6a4e82b6caf95841b07bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0746f8bb491a45df8d964c43c37dd59c"
I20240503 08:11:20.992036 30353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6f32fcd7b44255a18756ce7d2d44f2" candidate_uuid: "2a33001abc6a4e82b6caf95841b07bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f415ea6340664cc5ae70fa0db5095545"
I20240503 08:11:20.992347 30353 raft_consensus.cc:2379] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a33001abc6a4e82b6caf95841b07bf0 in current term 1: Already voted for candidate f415ea6340664cc5ae70fa0db5095545 in this term.
I20240503 08:11:20.992817 29832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6f32fcd7b44255a18756ce7d2d44f2" candidate_uuid: "f415ea6340664cc5ae70fa0db5095545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a33001abc6a4e82b6caf95841b07bf0"
I20240503 08:11:20.993104 29832 raft_consensus.cc:2379] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f415ea6340664cc5ae70fa0db5095545 in current term 1: Already voted for candidate 2a33001abc6a4e82b6caf95841b07bf0 in this term.
I20240503 08:11:20.993446 29766 leader_election.cc:302] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a33001abc6a4e82b6caf95841b07bf0; no voters: 0746f8bb491a45df8d964c43c37dd59c, f415ea6340664cc5ae70fa0db5095545
I20240503 08:11:20.993803 30422 raft_consensus.cc:2726] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240503 08:11:21.039803 30399 tablet.cc:2369] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:11:21.047003 30269 tablet.cc:2369] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:21.094349 30223 raft_consensus.cc:2454] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 0746f8bb491a45df8d964c43c37dd59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f415ea6340664cc5ae70fa0db5095545 in term 1.
I20240503 08:11:21.094899 30286 leader_election.cc:302] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0746f8bb491a45df8d964c43c37dd59c, f415ea6340664cc5ae70fa0db5095545; no voters: 2a33001abc6a4e82b6caf95841b07bf0
I20240503 08:11:21.095285 30423 raft_consensus.cc:2781] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:21.095849 30423 raft_consensus.cc:686] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [term 1 LEADER]: Becoming Leader. State: Replica: f415ea6340664cc5ae70fa0db5095545, State: Running, Role: LEADER
I20240503 08:11:21.096880 30423 consensus_queue.cc:237] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } }
I20240503 08:11:21.102636 29691 catalog_manager.cc:5484] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 reported cstate change: term changed from 0 to 1, leader changed from <none> to f415ea6340664cc5ae70fa0db5095545 (127.27.157.197). New cstate: current_term: 1 leader_uuid: "f415ea6340664cc5ae70fa0db5095545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a33001abc6a4e82b6caf95841b07bf0" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f415ea6340664cc5ae70fa0db5095545" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 46799 } health_report { overall_health: HEALTHY } } }
W20240503 08:11:21.111495 29878 tablet.cc:2369] T 5a6f32fcd7b44255a18756ce7d2d44f2 P 2a33001abc6a4e82b6caf95841b07bf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:11:21.156165 28279 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 08:11:21.158644 28279 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2a33001abc6a4e82b6caf95841b07bf0 to finish bootstrapping
I20240503 08:11:21.168741 28279 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5a2a2bcf493e4b88bb329e7411c70940 to finish bootstrapping
I20240503 08:11:21.178927 28279 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ee2c63f4e028410c99f5132d75489b53 to finish bootstrapping
I20240503 08:11:21.189612 28279 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0746f8bb491a45df8d964c43c37dd59c to finish bootstrapping
I20240503 08:11:21.199494 28279 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f415ea6340664cc5ae70fa0db5095545 to finish bootstrapping
I20240503 08:11:21.208119 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 29750
I20240503 08:11:21.209439 29872 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:21.300999 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 29750
W20240503 08:11:21.596233 30287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.157.193:39577: connect: Connection refused (error 111)
W20240503 08:11:21.601392 30287 consensus_peers.cc:485] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 -> Peer 2a33001abc6a4e82b6caf95841b07bf0 (127.27.157.193:39577): Couldn't send request to peer 2a33001abc6a4e82b6caf95841b07bf0. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:39577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:21.649822 30423 consensus_queue.cc:1035] T 5a6f32fcd7b44255a18756ce7d2d44f2 P f415ea6340664cc5ae70fa0db5095545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0746f8bb491a45df8d964c43c37dd59c" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:21.834398 30203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:21.835997 29943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:21.841068 30073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:21.843561 30333 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:21.984611 30203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:21.987579 29943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:21.988920 30073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:21.992795 30333 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:22.163169 30203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:22.165194 29943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:22.172381 30333 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:22.181658 30073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:22.834791 30203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:22.836488 29943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:22.842662 30073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:22.843559 30333 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:22.977075 30203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:22.978719 29943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:22.986083 30073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:22.990386 30333 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:23.169554 30203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:23.178369 29943 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:23.185560 30073 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:23.187067 30333 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:23.344669 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 29881
I20240503 08:11:23.349162 30003 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:23.442776 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 29881
W20240503 08:11:23.460665 29675 connection.cc:535] server connection from 127.27.157.194:59421 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:23.460980 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 30011
I20240503 08:11:23.462039 30133 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:23.557538 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 30011
I20240503 08:11:23.577594 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 30141
I20240503 08:11:23.579010 30263 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:23.680016 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 30141
W20240503 08:11:23.698127 29676 connection.cc:535] server connection from 127.27.157.196:37553 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:11:23.698398 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 30271
I20240503 08:11:23.699548 30393 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:23.800693 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 30271
I20240503 08:11:23.819299 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 29660
I20240503 08:11:23.820577 29721 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:23.939155 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 29660
2024-05-03T08:11:23Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (6210 ms)
[----------] 1 test from RebalanceStartCriteriaTest (6210 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240503 08:11:23.984817 28279 test_util.cc:268] Using random seed: 1140729750
I20240503 08:11:23.986850 28279 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:23.986922 28279 ts_itest-base.cc:113] --------------
I20240503 08:11:23.986966 28279 ts_itest-base.cc:114] 7 tablet servers
I20240503 08:11:23.987022 28279 ts_itest-base.cc:115] 1 replicas per TS
I20240503 08:11:23.987071 28279 ts_itest-base.cc:116] --------------
2024-05-03T08:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:23Z Disabled control of system clock
I20240503 08:11:24.014106 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.157.254:36149
--webserver_interface=127.27.157.254
--webserver_port=0
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.157.254:36149
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 08:11:24.365180 30638 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 08:11:24.365561 30638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:24.365630 30638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:24.365711 30638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:24.373901 30638 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:24.374047 30638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:24.374109 30638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:24.374169 30638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:24.374251 30638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:24.384550 30638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.157.254:36149
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.157.254:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.157.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:24.386261 30638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:24.388357 30638 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:24.400646 30643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:24.401425 30644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:24.402385 30646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:24.403460 30638 server_base.cc:1034] running on GCE node
I20240503 08:11:24.404356 30638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:24.405977 30638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:24.407248 30638 hybrid_clock.cc:648] HybridClock initialized: now 1714723884407155 us; error 71 us; skew 500 ppm
I20240503 08:11:24.407649 30638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:24.410436 30638 webserver.cc:460] Webserver started at http://127.27.157.254:43937/ using document root <none> and password file <none>
I20240503 08:11:24.411281 30638 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:24.411487 30638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:24.411818 30638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:24.414808 30638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d2cdaf8faf8c4440943fe8633faf60f7"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-dj8w"
I20240503 08:11:24.415859 30638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d2cdaf8faf8c4440943fe8633faf60f7"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-dj8w"
I20240503 08:11:24.422119 30638 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240503 08:11:24.426324 30652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:24.428285 30638 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20240503 08:11:24.428503 30638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d2cdaf8faf8c4440943fe8633faf60f7"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-dj8w"
I20240503 08:11:24.428727 30638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:24.456532 30638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:24.457485 30638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:24.457829 30638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:24.477205 30638 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.254:36149
I20240503 08:11:24.477226 30703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.254:36149 every 8 connection(s)
I20240503 08:11:24.478740 30638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:24.483023 30704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:24.485237 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 30638
I20240503 08:11:24.485448 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:24.496632 30704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7: Bootstrap starting.
I20240503 08:11:24.501070 30704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:24.502411 30704 log.cc:824] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:24.505614 30704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7: No bootstrap required, opened a new log
I20240503 08:11:24.512833 30704 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cdaf8faf8c4440943fe8633faf60f7" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 36149 } }
I20240503 08:11:24.513325 30704 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:24.513458 30704 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2cdaf8faf8c4440943fe8633faf60f7, State: Initialized, Role: FOLLOWER
I20240503 08:11:24.514214 30704 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cdaf8faf8c4440943fe8633faf60f7" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 36149 } }
I20240503 08:11:24.514456 30704 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:24.514590 30704 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:24.514763 30704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:24.517333 30704 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cdaf8faf8c4440943fe8633faf60f7" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 36149 } }
I20240503 08:11:24.518009 30704 leader_election.cc:302] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2cdaf8faf8c4440943fe8633faf60f7; no voters:
I20240503 08:11:24.518703 30704 leader_election.cc:288] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:24.518870 30709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:24.519590 30709 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [term 1 LEADER]: Becoming Leader. State: Replica: d2cdaf8faf8c4440943fe8633faf60f7, State: Running, Role: LEADER
I20240503 08:11:24.520223 30709 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cdaf8faf8c4440943fe8633faf60f7" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 36149 } }
I20240503 08:11:24.520931 30704 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:24.523370 30710 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2cdaf8faf8c4440943fe8633faf60f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cdaf8faf8c4440943fe8633faf60f7" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 36149 } } }
I20240503 08:11:24.523973 30710 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:24.523397 30711 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2cdaf8faf8c4440943fe8633faf60f7. Latest consensus state: current_term: 1 leader_uuid: "d2cdaf8faf8c4440943fe8633faf60f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2cdaf8faf8c4440943fe8633faf60f7" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 36149 } } }
I20240503 08:11:24.524776 30711 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7 [sys.catalog]: This master's current role is: LEADER
W20240503 08:11:24.537474 30725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:11:24.537659 30725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:11:24.537801 30717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:24.539419 30717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:24.548557 30717 catalog_manager.cc:1347] Generated new cluster ID: 5952856a5e77472caca0c959fecb35a0
I20240503 08:11:24.548708 30717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:24.560587 30717 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:24.561952 30717 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:24.577737 30717 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d2cdaf8faf8c4440943fe8633faf60f7: Generated new TSK 0
I20240503 08:11:24.578868 30717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:24.596426 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.193:0
--local_ip_for_outbound_sockets=127.27.157.193
--webserver_interface=127.27.157.193
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:24.949788 30728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:24.950109 30728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:24.950233 30728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:24.957940 30728 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:24.958083 30728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:24.958254 30728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.193
I20240503 08:11:24.968371 30728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.157.193
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.193
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:24.969753 30728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:24.971601 30728 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:24.985349 30736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:24.985844 30733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:24.986793 30734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:24.987705 30728 server_base.cc:1034] running on GCE node
I20240503 08:11:24.988530 30728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:24.989727 30728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:24.990978 30728 hybrid_clock.cc:648] HybridClock initialized: now 1714723884990937 us; error 35 us; skew 500 ppm
I20240503 08:11:24.991369 30728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:24.993893 30728 webserver.cc:460] Webserver started at http://127.27.157.193:37149/ using document root <none> and password file <none>
I20240503 08:11:24.994695 30728 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:24.994814 30728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:24.995134 30728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:24.998026 30728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3986faa56d004ea5b787a469ddb79c81"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-dj8w"
I20240503 08:11:24.998826 30728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3986faa56d004ea5b787a469ddb79c81"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-dj8w"
I20240503 08:11:25.004681 30728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240503 08:11:25.008550 30742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.010025 30728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 08:11:25.010196 30728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3986faa56d004ea5b787a469ddb79c81"
format_stamp: "Formatted at 2024-05-03 08:11:24 on dist-test-slave-dj8w"
I20240503 08:11:25.010419 30728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:25.034904 30728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:25.035867 30728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:25.036152 30728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:25.037221 30728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:25.038866 30728 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:25.038964 30728 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.039083 30728 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:25.039180 30728 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.073498 30728 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.193:44943
I20240503 08:11:25.073555 30854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.193:44943 every 8 connection(s)
I20240503 08:11:25.074923 30728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:25.078372 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 30728
I20240503 08:11:25.078629 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:25.082538 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.194:0
--local_ip_for_outbound_sockets=127.27.157.194
--webserver_interface=127.27.157.194
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:25.091375 30855 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:25.091761 30855 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:25.092916 30855 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:25.095786 30669 ts_manager.cc:194] Registered new tserver with Master: 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:25.098307 30669 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.193:52245
W20240503 08:11:25.434407 30859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:25.434760 30859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:25.434921 30859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:25.442638 30859 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:25.442821 30859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:25.442989 30859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.194
I20240503 08:11:25.452502 30859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.157.194
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.194
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:25.453958 30859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:25.455900 30859 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:25.469012 30864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.469630 30865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.470563 30867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:25.471436 30859 server_base.cc:1034] running on GCE node
I20240503 08:11:25.472139 30859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:25.473284 30859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:25.474562 30859 hybrid_clock.cc:648] HybridClock initialized: now 1714723885474504 us; error 47 us; skew 500 ppm
I20240503 08:11:25.474965 30859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:25.477576 30859 webserver.cc:460] Webserver started at http://127.27.157.194:35869/ using document root <none> and password file <none>
I20240503 08:11:25.478346 30859 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:25.478473 30859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:25.478814 30859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:25.481685 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ac3512789abc4bd890102fa6910f980b"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-dj8w"
I20240503 08:11:25.482532 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ac3512789abc4bd890102fa6910f980b"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-dj8w"
I20240503 08:11:25.489208 30859 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240503 08:11:25.493186 30873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.494721 30859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 08:11:25.494933 30859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ac3512789abc4bd890102fa6910f980b"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-dj8w"
I20240503 08:11:25.495164 30859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:25.538811 30859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:25.539748 30859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:25.540046 30859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:25.541110 30859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:25.542706 30859 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:25.542821 30859 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.542938 30859 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:25.543013 30859 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:25.576622 30859 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.194:36429
I20240503 08:11:25.576716 30985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.194:36429 every 8 connection(s)
I20240503 08:11:25.578189 30859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:25.585492 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 30859
I20240503 08:11:25.585687 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:25.589048 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.195:0
--local_ip_for_outbound_sockets=127.27.157.195
--webserver_interface=127.27.157.195
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:25.592770 30986 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:25.593168 30986 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:25.594113 30986 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:25.595686 30669 ts_manager.cc:194] Registered new tserver with Master: ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429)
I20240503 08:11:25.596511 30669 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.194:39075
W20240503 08:11:25.942077 30989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:25.942396 30989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:25.942521 30989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:25.950461 30989 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:25.950604 30989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:25.950775 30989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.195
I20240503 08:11:25.960294 30989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.157.195
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.195
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:25.961727 30989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:25.963635 30989 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:25.976823 30995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.977609 30997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:25.978312 30994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:25.979204 30989 server_base.cc:1034] running on GCE node
I20240503 08:11:25.979986 30989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:25.981082 30989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:25.982335 30989 hybrid_clock.cc:648] HybridClock initialized: now 1714723885982286 us; error 51 us; skew 500 ppm
I20240503 08:11:25.982666 30989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:25.985277 30989 webserver.cc:460] Webserver started at http://127.27.157.195:36979/ using document root <none> and password file <none>
I20240503 08:11:25.986081 30989 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:25.986187 30989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:25.986474 30989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:25.989317 30989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a9b83be26ed14471b0335ffb28f12193"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-dj8w"
I20240503 08:11:25.990110 30989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a9b83be26ed14471b0335ffb28f12193"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-dj8w"
I20240503 08:11:25.995849 30989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20240503 08:11:25.999784 31003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:26.001546 30989 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:11:26.001724 30989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a9b83be26ed14471b0335ffb28f12193"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-dj8w"
I20240503 08:11:26.001982 30989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:26.018697 30989 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:26.019683 30989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:26.019979 30989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:26.021052 30989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:26.022785 30989 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:26.022889 30989 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:26.023013 30989 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:26.023080 30989 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:26.058012 30989 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.195:42421
I20240503 08:11:26.058066 31115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.195:42421 every 8 connection(s)
I20240503 08:11:26.059464 30989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:26.059728 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 30989
I20240503 08:11:26.059921 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:26.063602 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.196:0
--local_ip_for_outbound_sockets=127.27.157.196
--webserver_interface=127.27.157.196
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:26.076824 31116 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:26.077258 31116 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:26.078562 31116 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:26.081086 30669 ts_manager.cc:194] Registered new tserver with Master: a9b83be26ed14471b0335ffb28f12193 (127.27.157.195:42421)
I20240503 08:11:26.082181 30669 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.195:56097
I20240503 08:11:26.102905 30855 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
W20240503 08:11:26.419245 31119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:26.419633 31119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:26.419791 31119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:26.427721 31119 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:26.427896 31119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:26.428067 31119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.196
I20240503 08:11:26.437793 31119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.157.196
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.196
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:26.439260 31119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:26.441156 31119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:26.454810 31124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:26.455724 31125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:26.456395 31127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:26.457388 31119 server_base.cc:1034] running on GCE node
I20240503 08:11:26.458062 31119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:26.459178 31119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:26.460420 31119 hybrid_clock.cc:648] HybridClock initialized: now 1714723886460379 us; error 45 us; skew 500 ppm
I20240503 08:11:26.460822 31119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:26.463397 31119 webserver.cc:460] Webserver started at http://127.27.157.196:46165/ using document root <none> and password file <none>
I20240503 08:11:26.464226 31119 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:26.464354 31119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:26.464679 31119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:26.467485 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a7b227b516a24a1b837415111eba1303"
format_stamp: "Formatted at 2024-05-03 08:11:26 on dist-test-slave-dj8w"
I20240503 08:11:26.468327 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a7b227b516a24a1b837415111eba1303"
format_stamp: "Formatted at 2024-05-03 08:11:26 on dist-test-slave-dj8w"
I20240503 08:11:26.473990 31119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:26.478019 31133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:26.479972 31119 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:26.480190 31119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a7b227b516a24a1b837415111eba1303"
format_stamp: "Formatted at 2024-05-03 08:11:26 on dist-test-slave-dj8w"
I20240503 08:11:26.480443 31119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:26.496357 31119 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:26.497208 31119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:26.497473 31119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:26.498437 31119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:26.500176 31119 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:26.500276 31119 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:26.500387 31119 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:26.500458 31119 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:26.535315 31119 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.196:42725
I20240503 08:11:26.535426 31245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.196:42725 every 8 connection(s)
I20240503 08:11:26.536799 31119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:26.537214 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 31119
I20240503 08:11:26.537392 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:26.541208 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.197:0
--local_ip_for_outbound_sockets=127.27.157.197
--webserver_interface=127.27.157.197
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:26.553339 31246 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:26.553678 31246 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:26.554639 31246 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:26.556360 30669 ts_manager.cc:194] Registered new tserver with Master: a7b227b516a24a1b837415111eba1303 (127.27.157.196:42725)
I20240503 08:11:26.557183 30669 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.196:50071
I20240503 08:11:26.599429 30986 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
W20240503 08:11:26.893460 31249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:26.893792 31249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:26.893887 31249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:26.901635 31249 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:26.901780 31249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:26.901894 31249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.197
I20240503 08:11:26.911674 31249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.157.197
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.197
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:26.913100 31249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:26.915040 31249 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:26.928802 31254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:26.929600 31257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:26.930573 31255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:26.931488 31249 server_base.cc:1034] running on GCE node
I20240503 08:11:26.932323 31249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:26.933434 31249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:26.934677 31249 hybrid_clock.cc:648] HybridClock initialized: now 1714723886934608 us; error 64 us; skew 500 ppm
I20240503 08:11:26.935051 31249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:26.942178 31249 webserver.cc:460] Webserver started at http://127.27.157.197:38913/ using document root <none> and password file <none>
I20240503 08:11:26.943009 31249 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:26.943146 31249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:26.943567 31249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:26.946388 31249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c5dc2c8f44d44889b3888f6265d07de9"
format_stamp: "Formatted at 2024-05-03 08:11:26 on dist-test-slave-dj8w"
I20240503 08:11:26.947238 31249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c5dc2c8f44d44889b3888f6265d07de9"
format_stamp: "Formatted at 2024-05-03 08:11:26 on dist-test-slave-dj8w"
I20240503 08:11:26.953644 31249 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20240503 08:11:26.957865 31263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:26.959636 31249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:26.959872 31249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c5dc2c8f44d44889b3888f6265d07de9"
format_stamp: "Formatted at 2024-05-03 08:11:26 on dist-test-slave-dj8w"
I20240503 08:11:26.960119 31249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:26.981842 31249 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:26.982808 31249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:26.983120 31249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:26.984236 31249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:26.985873 31249 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:26.985975 31249 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:26.986095 31249 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:26.986189 31249 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:27.019908 31249 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.197:34939
I20240503 08:11:27.019984 31375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.197:34939 every 8 connection(s)
I20240503 08:11:27.021275 31249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:27.024184 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 31249
I20240503 08:11:27.024397 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:27.027493 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.198:0
--local_ip_for_outbound_sockets=127.27.157.198
--webserver_interface=127.27.157.198
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:27.036134 31376 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:27.036474 31376 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:27.037386 31376 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:27.039023 30669 ts_manager.cc:194] Registered new tserver with Master: c5dc2c8f44d44889b3888f6265d07de9 (127.27.157.197:34939)
I20240503 08:11:27.040341 30669 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.197:57447
I20240503 08:11:27.086458 31116 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
W20240503 08:11:27.381165 31379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:27.381510 31379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:27.381628 31379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:27.389751 31379 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:27.389899 31379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:27.390074 31379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.198
I20240503 08:11:27.399574 31379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.157.198
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.198
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:27.401052 31379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:27.402977 31379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:27.417309 31384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:27.417722 31387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:27.418820 31385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:27.419747 31379 server_base.cc:1034] running on GCE node
I20240503 08:11:27.420493 31379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:27.421808 31379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:27.423066 31379 hybrid_clock.cc:648] HybridClock initialized: now 1714723887423007 us; error 51 us; skew 500 ppm
I20240503 08:11:27.423518 31379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:27.426215 31379 webserver.cc:460] Webserver started at http://127.27.157.198:46543/ using document root <none> and password file <none>
I20240503 08:11:27.427078 31379 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:27.427263 31379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:27.427623 31379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:27.430668 31379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9f45625e8fff471382d1376ff8cf0d00"
format_stamp: "Formatted at 2024-05-03 08:11:27 on dist-test-slave-dj8w"
I20240503 08:11:27.431594 31379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9f45625e8fff471382d1376ff8cf0d00"
format_stamp: "Formatted at 2024-05-03 08:11:27 on dist-test-slave-dj8w"
I20240503 08:11:27.437762 31379 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240503 08:11:27.441838 31393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:27.443564 31379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:27.443781 31379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9f45625e8fff471382d1376ff8cf0d00"
format_stamp: "Formatted at 2024-05-03 08:11:27 on dist-test-slave-dj8w"
I20240503 08:11:27.444028 31379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:27.471127 31379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:27.472155 31379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:27.472494 31379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:27.473601 31379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:27.475363 31379 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:27.475477 31379 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20240503 08:11:27.475597 31379 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:27.475678 31379 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:27.511301 31379 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.198:42339
I20240503 08:11:27.511422 31505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.198:42339 every 8 connection(s)
I20240503 08:11:27.512912 31379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:11:27.519510 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 31379
I20240503 08:11:27.519712 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:11:27.523626 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.199:0
--local_ip_for_outbound_sockets=127.27.157.199
--webserver_interface=127.27.157.199
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:27.529376 31506 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:27.529750 31506 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:27.530767 31506 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:27.532733 30669 ts_manager.cc:194] Registered new tserver with Master: 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198:42339)
I20240503 08:11:27.533722 30669 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.198:35865
I20240503 08:11:27.560022 31246 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
W20240503 08:11:27.888245 31509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:27.888563 31509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:27.888680 31509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:27.896744 31509 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:27.896926 31509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:27.897121 31509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.199
I20240503 08:11:27.907090 31509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.157.199
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.199
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:27.908576 31509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:27.910490 31509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:27.925362 31515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:27.926031 31517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:27.926594 31514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:27.927662 31509 server_base.cc:1034] running on GCE node
I20240503 08:11:27.928388 31509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:27.929478 31509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:27.930721 31509 hybrid_clock.cc:648] HybridClock initialized: now 1714723887930660 us; error 42 us; skew 500 ppm
I20240503 08:11:27.931075 31509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:27.933903 31509 webserver.cc:460] Webserver started at http://127.27.157.199:33489/ using document root <none> and password file <none>
I20240503 08:11:27.934684 31509 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:27.934818 31509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:27.935148 31509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:27.938011 31509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "03e83fbce8ee4c5cacbf54f25e1f2665"
format_stamp: "Formatted at 2024-05-03 08:11:27 on dist-test-slave-dj8w"
I20240503 08:11:27.938827 31509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "03e83fbce8ee4c5cacbf54f25e1f2665"
format_stamp: "Formatted at 2024-05-03 08:11:27 on dist-test-slave-dj8w"
I20240503 08:11:27.944751 31509 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240503 08:11:27.948926 31523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:27.950579 31509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:11:27.950758 31509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "03e83fbce8ee4c5cacbf54f25e1f2665"
format_stamp: "Formatted at 2024-05-03 08:11:27 on dist-test-slave-dj8w"
I20240503 08:11:27.950969 31509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:27.969676 31509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:27.970583 31509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:27.970853 31509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:27.971861 31509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:27.973449 31509 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:27.973536 31509 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:27.973624 31509 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:27.973680 31509 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:28.008416 31509 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.199:36035
I20240503 08:11:28.008464 31635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.199:36035 every 8 connection(s)
I20240503 08:11:28.009847 31509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:11:28.018100 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 31509
I20240503 08:11:28.018328 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:11:28.024626 31636 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:28.024991 31636 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:28.025907 31636 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:28.027987 30669 ts_manager.cc:194] Registered new tserver with Master: 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199:36035)
I20240503 08:11:28.028882 30669 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.199:46155
I20240503 08:11:28.036762 28279 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 08:11:28.043272 31376 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
I20240503 08:11:28.056905 30669 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56466:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 08:11:28.088213 31311 tablet_service.cc:1467] Processing CreateTablet for tablet ded8fc9e27bf4613bf2f3dd4e9db250a (DEFAULT_TABLE table=TestTable [id=5b61897243314d789557878679365bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:28.089814 31311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded8fc9e27bf4613bf2f3dd4e9db250a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:28.101220 31650 tablet_bootstrap.cc:492] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: Bootstrap starting.
I20240503 08:11:28.104660 31650 tablet_bootstrap.cc:654] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:28.105741 31650 log.cc:824] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:28.108886 31650 tablet_bootstrap.cc:492] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: No bootstrap required, opened a new log
I20240503 08:11:28.109220 31650 ts_tablet_manager.cc:1367] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 08:11:28.115752 31650 raft_consensus.cc:348] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } }
I20240503 08:11:28.116184 31650 raft_consensus.cc:374] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:28.116300 31650 raft_consensus.cc:729] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5dc2c8f44d44889b3888f6265d07de9, State: Initialized, Role: FOLLOWER
I20240503 08:11:28.116973 31650 consensus_queue.cc:260] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } }
I20240503 08:11:28.117182 31650 raft_consensus.cc:388] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:28.117314 31650 raft_consensus.cc:482] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:28.117475 31650 raft_consensus.cc:3037] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:28.120085 31650 raft_consensus.cc:504] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } }
I20240503 08:11:28.120672 31650 leader_election.cc:302] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5dc2c8f44d44889b3888f6265d07de9; no voters:
I20240503 08:11:28.121299 31650 leader_election.cc:288] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:28.121480 31652 raft_consensus.cc:2781] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:28.122247 31652 raft_consensus.cc:686] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 1 LEADER]: Becoming Leader. State: Replica: c5dc2c8f44d44889b3888f6265d07de9, State: Running, Role: LEADER
I20240503 08:11:28.122848 31652 consensus_queue.cc:237] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } }
I20240503 08:11:28.124593 31650 ts_tablet_manager.cc:1398] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20240503 08:11:28.129604 30669 catalog_manager.cc:5484] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5dc2c8f44d44889b3888f6265d07de9 (127.27.157.197). New cstate: current_term: 1 leader_uuid: "c5dc2c8f44d44889b3888f6265d07de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:28.149391 28279 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 08:11:28.151765 28279 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3986faa56d004ea5b787a469ddb79c81 to finish bootstrapping
I20240503 08:11:28.162438 28279 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ac3512789abc4bd890102fa6910f980b to finish bootstrapping
I20240503 08:11:28.172195 28279 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a9b83be26ed14471b0335ffb28f12193 to finish bootstrapping
I20240503 08:11:28.182035 28279 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a7b227b516a24a1b837415111eba1303 to finish bootstrapping
I20240503 08:11:28.192080 28279 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c5dc2c8f44d44889b3888f6265d07de9 to finish bootstrapping
I20240503 08:11:28.200369 28279 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9f45625e8fff471382d1376ff8cf0d00 to finish bootstrapping
I20240503 08:11:28.210183 28279 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 03e83fbce8ee4c5cacbf54f25e1f2665 to finish bootstrapping
I20240503 08:11:28.220072 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 30989
I20240503 08:11:28.221330 31111 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:28.308600 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 30989
I20240503 08:11:28.325287 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 31119
I20240503 08:11:28.326499 31241 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:28.413395 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 31119
I20240503 08:11:28.430155 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 31249
I20240503 08:11:28.431351 31371 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:28.529666 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 31249
I20240503 08:11:28.537057 31506 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
I20240503 08:11:28.546826 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 31379
I20240503 08:11:28.548148 31501 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:28.636660 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 31379
I20240503 08:11:28.655881 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 31509
I20240503 08:11:28.657030 31631 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:28.740157 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 31509
I20240503 08:11:32.508347 30669 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56466:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:32.541311 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4d7f23456f4bddabcdd74ddea40a33 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b355892f4af64661884a4ba55276ce1c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.541277 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 87f2891ba60d452483f1edaf2a7f9eb3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b355892f4af64661884a4ba55276ce1c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.541335 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 74ae8557f32444f6b60487f75b6e353a (DEFAULT_TABLE table=rebalance_test_table_0 [id=b355892f4af64661884a4ba55276ce1c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:32.543064 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4d7f23456f4bddabcdd74ddea40a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.543071 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87f2891ba60d452483f1edaf2a7f9eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.543646 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ae8557f32444f6b60487f75b6e353a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:32.556275 31677 tablet_bootstrap.cc:492] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81: Bootstrap starting.
I20240503 08:11:32.556890 31678 tablet_bootstrap.cc:492] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b: Bootstrap starting.
I20240503 08:11:32.561091 31677 tablet_bootstrap.cc:654] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.561093 31678 tablet_bootstrap.cc:654] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.562651 31677 log.cc:824] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:32.562651 31678 log.cc:824] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:32.566290 31678 tablet_bootstrap.cc:492] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b: No bootstrap required, opened a new log
I20240503 08:11:32.566291 31677 tablet_bootstrap.cc:492] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81: No bootstrap required, opened a new log
I20240503 08:11:32.566646 31677 ts_tablet_manager.cc:1367] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20240503 08:11:32.566646 31678 ts_tablet_manager.cc:1367] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240503 08:11:32.576912 31678 raft_consensus.cc:348] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:32.576910 31677 raft_consensus.cc:348] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:32.577440 31677 raft_consensus.cc:374] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.577445 31678 raft_consensus.cc:374] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.577579 31678 raft_consensus.cc:729] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.577598 31677 raft_consensus.cc:729] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.578462 31677 consensus_queue.cc:260] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:32.578472 31678 consensus_queue.cc:260] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:32.578765 31677 raft_consensus.cc:388] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:32.578776 31678 raft_consensus.cc:388] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:32.578943 31678 raft_consensus.cc:482] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:32.578933 31677 raft_consensus.cc:482] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:32.579134 31677 raft_consensus.cc:3037] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.579140 31678 raft_consensus.cc:3037] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.582465 31678 raft_consensus.cc:504] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:32.582458 31677 raft_consensus.cc:504] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:32.583199 31678 leader_election.cc:302] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:32.583186 31677 leader_election.cc:302] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81; no voters:
I20240503 08:11:32.584106 31678 leader_election.cc:288] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:32.584226 31682 raft_consensus.cc:2781] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:32.584672 31677 leader_election.cc:288] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:32.584785 31681 raft_consensus.cc:2781] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:32.587989 31681 raft_consensus.cc:686] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Leader. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:32.588758 31681 consensus_queue.cc:237] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:32.591415 31682 raft_consensus.cc:686] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Leader. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:32.592078 31682 consensus_queue.cc:237] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:32.593487 31678 ts_tablet_manager.cc:1398] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b: Time spent starting tablet: real 0.027s user 0.021s sys 0.006s
I20240503 08:11:32.595760 31677 ts_tablet_manager.cc:1398] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81: Time spent starting tablet: real 0.029s user 0.021s sys 0.006s
I20240503 08:11:32.595983 30669 catalog_manager.cc:5484] T 74ae8557f32444f6b60487f75b6e353a P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193). New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:32.596464 31677 tablet_bootstrap.cc:492] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: Bootstrap starting.
I20240503 08:11:32.599112 31677 tablet_bootstrap.cc:654] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:32.601622 31677 tablet_bootstrap.cc:492] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: No bootstrap required, opened a new log
I20240503 08:11:32.601799 31677 ts_tablet_manager.cc:1367] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:11:32.602840 31677 raft_consensus.cc:348] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:32.603027 31677 raft_consensus.cc:374] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:32.603118 31677 raft_consensus.cc:729] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Initialized, Role: FOLLOWER
I20240503 08:11:32.603499 31677 consensus_queue.cc:260] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:32.603708 31677 raft_consensus.cc:388] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:32.603792 31677 raft_consensus.cc:482] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:32.603894 31677 raft_consensus.cc:3037] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:32.606349 30668 catalog_manager.cc:5484] T 87f2891ba60d452483f1edaf2a7f9eb3 P ac3512789abc4bd890102fa6910f980b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3512789abc4bd890102fa6910f980b (127.27.157.194). New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:32.606739 31677 raft_consensus.cc:504] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:32.607003 31677 leader_election.cc:302] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81; no voters:
I20240503 08:11:32.607299 31681 raft_consensus.cc:2781] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:32.607620 31681 raft_consensus.cc:686] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Leader. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:32.607852 31681 consensus_queue.cc:237] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:32.609935 31677 leader_election.cc:288] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:32.610790 31677 ts_tablet_manager.cc:1398] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240503 08:11:32.618664 30668 catalog_manager.cc:5484] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193). New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } health_report { overall_health: HEALTHY } } }
W20240503 08:11:32.968278 31693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:32.968613 31693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:32.976151 31693 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 139.946 ms
time per row: 0.069973 ms
I20240503 08:11:33.269785 30666 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56466:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:33.278282 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 20b5e4e7b78f4f0d979b936d92dd1eb3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f1bef55b064e9783e456fbd220acb4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:33.278393 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 6e188dd3547440b183799c9a796b4a85 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f1bef55b064e9783e456fbd220acb4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:33.278689 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0ad95a564241cf8a0e5c674e920a0b (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f1bef55b064e9783e456fbd220acb4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:33.278898 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b5e4e7b78f4f0d979b936d92dd1eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:33.279148 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0ad95a564241cf8a0e5c674e920a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:33.279170 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e188dd3547440b183799c9a796b4a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:33.284775 31737 tablet_bootstrap.cc:492] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: Bootstrap starting.
I20240503 08:11:33.286701 31737 tablet_bootstrap.cc:654] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:33.288654 31737 tablet_bootstrap.cc:492] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: No bootstrap required, opened a new log
I20240503 08:11:33.288800 31737 ts_tablet_manager.cc:1367] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 08:11:33.289659 31737 raft_consensus.cc:348] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.289820 31737 raft_consensus.cc:374] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:33.289917 31737 raft_consensus.cc:729] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Initialized, Role: FOLLOWER
I20240503 08:11:33.290156 31737 consensus_queue.cc:260] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.290326 31737 raft_consensus.cc:388] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:33.290408 31737 raft_consensus.cc:482] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:33.290513 31737 raft_consensus.cc:3037] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.292557 31737 raft_consensus.cc:504] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.292794 31737 leader_election.cc:302] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:33.292980 31737 leader_election.cc:288] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:33.293115 31727 raft_consensus.cc:2781] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:33.293723 31727 raft_consensus.cc:686] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Leader. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:33.293926 31738 tablet_bootstrap.cc:492] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81: Bootstrap starting.
I20240503 08:11:33.294176 31727 consensus_queue.cc:237] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.295642 31738 tablet_bootstrap.cc:654] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:33.297899 31737 ts_tablet_manager.cc:1398] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: Time spent starting tablet: real 0.009s user 0.003s sys 0.006s
I20240503 08:11:33.297830 30666 catalog_manager.cc:5484] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3512789abc4bd890102fa6910f980b (127.27.157.194). New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:33.298164 31737 tablet_bootstrap.cc:492] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: Bootstrap starting.
I20240503 08:11:33.298333 31738 tablet_bootstrap.cc:492] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81: No bootstrap required, opened a new log
I20240503 08:11:33.298488 31738 ts_tablet_manager.cc:1367] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 08:11:33.299302 31738 raft_consensus.cc:348] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:33.299571 31738 raft_consensus.cc:374] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:33.299671 31738 raft_consensus.cc:729] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Initialized, Role: FOLLOWER
I20240503 08:11:33.299947 31738 consensus_queue.cc:260] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:33.300119 31737 tablet_bootstrap.cc:654] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:33.300118 31738 raft_consensus.cc:388] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:33.300197 31738 raft_consensus.cc:482] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:33.300298 31738 raft_consensus.cc:3037] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.302070 31737 tablet_bootstrap.cc:492] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: No bootstrap required, opened a new log
I20240503 08:11:33.302239 31737 ts_tablet_manager.cc:1367] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:33.302697 31738 raft_consensus.cc:504] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:33.302984 31738 leader_election.cc:302] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81; no voters:
I20240503 08:11:33.303071 31737 raft_consensus.cc:348] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.303234 31738 leader_election.cc:288] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:33.303263 31737 raft_consensus.cc:374] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:33.303355 31737 raft_consensus.cc:729] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Initialized, Role: FOLLOWER
I20240503 08:11:33.303436 31729 raft_consensus.cc:2781] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:33.303607 31729 raft_consensus.cc:686] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Leader. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:33.303629 31737 consensus_queue.cc:260] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.303799 31737 raft_consensus.cc:388] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:33.303889 31737 raft_consensus.cc:482] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:33.303990 31737 raft_consensus.cc:3037] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.303943 31729 consensus_queue.cc:237] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:33.303920 31738 ts_tablet_manager.cc:1398] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20240503 08:11:33.306344 30667 catalog_manager.cc:5484] T 1d0ad95a564241cf8a0e5c674e920a0b P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193). New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:33.306509 31737 raft_consensus.cc:504] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.306736 31737 leader_election.cc:302] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:33.307071 31740 raft_consensus.cc:2781] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:33.307216 31740 raft_consensus.cc:686] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Leader. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:33.307538 31740 consensus_queue.cc:237] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.308727 31737 leader_election.cc:288] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:33.309351 31737 ts_tablet_manager.cc:1398] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20240503 08:11:33.310724 30667 catalog_manager.cc:5484] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3512789abc4bd890102fa6910f980b (127.27.157.194). New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } health_report { overall_health: HEALTHY } } }
W20240503 08:11:33.665235 31747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:33.665649 31747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:33.673599 31747 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 133.249 ms
time per row: 0.0666246 ms
I20240503 08:11:33.955431 30666 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56466:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:33.964587 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 153c5b78de3342eab81fdce1a7f199d5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=48d5c85892af46148fda788532045ac1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:33.964745 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 64aacda851904a08b428c85b2842e126 (DEFAULT_TABLE table=rebalance_test_table_2 [id=48d5c85892af46148fda788532045ac1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:33.964979 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 6b123f5c42af4d11b4325b3cd6226c29 (DEFAULT_TABLE table=rebalance_test_table_2 [id=48d5c85892af46148fda788532045ac1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:33.965144 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153c5b78de3342eab81fdce1a7f199d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:33.965219 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64aacda851904a08b428c85b2842e126. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:33.965411 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b123f5c42af4d11b4325b3cd6226c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:33.972309 31792 tablet_bootstrap.cc:492] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: Bootstrap starting.
I20240503 08:11:33.973050 31793 tablet_bootstrap.cc:492] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: Bootstrap starting.
I20240503 08:11:33.974105 31792 tablet_bootstrap.cc:654] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:33.974542 31793 tablet_bootstrap.cc:654] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:33.976161 31792 tablet_bootstrap.cc:492] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: No bootstrap required, opened a new log
I20240503 08:11:33.976343 31792 ts_tablet_manager.cc:1367] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:33.976681 31793 tablet_bootstrap.cc:492] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: No bootstrap required, opened a new log
I20240503 08:11:33.976843 31793 ts_tablet_manager.cc:1367] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:11:33.977248 31792 raft_consensus.cc:348] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:33.977450 31792 raft_consensus.cc:374] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:33.977552 31792 raft_consensus.cc:729] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Initialized, Role: FOLLOWER
I20240503 08:11:33.977644 31793 raft_consensus.cc:348] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.977836 31793 raft_consensus.cc:374] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:33.977825 31792 consensus_queue.cc:260] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:33.977932 31793 raft_consensus.cc:729] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Initialized, Role: FOLLOWER
I20240503 08:11:33.978037 31792 raft_consensus.cc:388] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:33.978123 31792 raft_consensus.cc:482] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:33.978248 31792 raft_consensus.cc:3037] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.978180 31793 consensus_queue.cc:260] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.978389 31793 raft_consensus.cc:388] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:33.978471 31793 raft_consensus.cc:482] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:33.978600 31793 raft_consensus.cc:3037] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.981338 31792 raft_consensus.cc:504] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:33.981506 31793 raft_consensus.cc:504] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.981683 31792 leader_election.cc:302] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81; no voters:
I20240503 08:11:33.981762 31793 leader_election.cc:302] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:33.981933 31792 leader_election.cc:288] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:33.981956 31793 leader_election.cc:288] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:33.982065 31785 raft_consensus.cc:2781] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:33.982055 31740 raft_consensus.cc:2781] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:33.982460 31793 ts_tablet_manager.cc:1398] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:11:33.983876 31740 raft_consensus.cc:686] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Leader. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:33.984171 31740 consensus_queue.cc:237] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:33.985006 31785 raft_consensus.cc:686] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Leader. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:33.985379 31785 consensus_queue.cc:237] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:33.987753 30666 catalog_manager.cc:5484] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3512789abc4bd890102fa6910f980b (127.27.157.194). New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:33.988145 31792 ts_tablet_manager.cc:1398] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20240503 08:11:33.988471 31792 tablet_bootstrap.cc:492] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: Bootstrap starting.
I20240503 08:11:33.989475 30667 catalog_manager.cc:5484] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193). New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:33.990947 31792 tablet_bootstrap.cc:654] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:33.994000 31792 tablet_bootstrap.cc:492] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: No bootstrap required, opened a new log
I20240503 08:11:33.994200 31792 ts_tablet_manager.cc:1367] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 08:11:33.995069 31792 raft_consensus.cc:348] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:33.995280 31792 raft_consensus.cc:374] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:33.995398 31792 raft_consensus.cc:729] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Initialized, Role: FOLLOWER
I20240503 08:11:33.995642 31792 consensus_queue.cc:260] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:33.995810 31792 raft_consensus.cc:388] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:33.995909 31792 raft_consensus.cc:482] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:33.996001 31792 raft_consensus.cc:3037] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:33.998076 31792 raft_consensus.cc:504] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:33.998317 31792 leader_election.cc:302] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81; no voters:
I20240503 08:11:33.998524 31792 leader_election.cc:288] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:33.998672 31798 raft_consensus.cc:2781] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:33.999015 31792 ts_tablet_manager.cc:1398] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:11:33.999258 31798 raft_consensus.cc:686] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Leader. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:33.999619 31798 consensus_queue.cc:237] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:34.002143 30667 catalog_manager.cc:5484] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193). New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } health_report { overall_health: HEALTHY } } }
W20240503 08:11:34.351898 31803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:34.352293 31803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:34.360190 31803 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 162.114 ms
time per row: 0.081057 ms
I20240503 08:11:34.677932 30660 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56466:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:34.687314 30920 tablet_service.cc:1467] Processing CreateTablet for tablet fa28377977cf4ffa9347b5d45020c377 (DEFAULT_TABLE table=rebalance_test_table_3 [id=4074a9f3638e495cb809de913789b611]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.687590 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5f4bb2e73c4643a964b3c834b820aa (DEFAULT_TABLE table=rebalance_test_table_3 [id=4074a9f3638e495cb809de913789b611]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.687690 30921 tablet_service.cc:1467] Processing CreateTablet for tablet efe80a3ea5154c3a86d3816ba79657ce (DEFAULT_TABLE table=rebalance_test_table_3 [id=4074a9f3638e495cb809de913789b611]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:34.687892 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa28377977cf4ffa9347b5d45020c377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.688046 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5f4bb2e73c4643a964b3c834b820aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.688158 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe80a3ea5154c3a86d3816ba79657ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:34.694769 31850 tablet_bootstrap.cc:492] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: Bootstrap starting.
I20240503 08:11:34.696333 31850 tablet_bootstrap.cc:654] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.698323 31850 tablet_bootstrap.cc:492] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: No bootstrap required, opened a new log
I20240503 08:11:34.698483 31850 ts_tablet_manager.cc:1367] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:34.699136 31850 raft_consensus.cc:348] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:34.699311 31850 raft_consensus.cc:374] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.699443 31850 raft_consensus.cc:729] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.699676 31850 consensus_queue.cc:260] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:34.699862 31850 raft_consensus.cc:388] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:34.699934 31850 raft_consensus.cc:482] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:34.700058 31850 raft_consensus.cc:3037] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.701978 31850 raft_consensus.cc:504] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:34.702244 31850 leader_election.cc:302] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:34.702477 31850 leader_election.cc:288] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:34.702596 31837 raft_consensus.cc:2781] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:34.703198 31851 tablet_bootstrap.cc:492] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: Bootstrap starting.
I20240503 08:11:34.704820 31851 tablet_bootstrap.cc:654] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.706081 31850 ts_tablet_manager.cc:1398] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20240503 08:11:34.706383 31850 tablet_bootstrap.cc:492] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b: Bootstrap starting.
I20240503 08:11:34.706522 31837 raft_consensus.cc:686] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Leader. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:34.706849 31837 consensus_queue.cc:237] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:34.708210 31850 tablet_bootstrap.cc:654] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:34.708544 31851 tablet_bootstrap.cc:492] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: No bootstrap required, opened a new log
I20240503 08:11:34.708778 31851 ts_tablet_manager.cc:1367] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:11:34.709892 31851 raft_consensus.cc:348] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:34.710253 31851 raft_consensus.cc:374] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.710424 31851 raft_consensus.cc:729] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.710803 31851 consensus_queue.cc:260] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:34.710981 31851 raft_consensus.cc:388] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:34.711081 31851 raft_consensus.cc:482] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:34.711210 31851 raft_consensus.cc:3037] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.711234 30660 catalog_manager.cc:5484] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3512789abc4bd890102fa6910f980b (127.27.157.194). New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:34.711791 31850 tablet_bootstrap.cc:492] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b: No bootstrap required, opened a new log
I20240503 08:11:34.711970 31850 ts_tablet_manager.cc:1367] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 08:11:34.712837 31850 raft_consensus.cc:348] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:34.713050 31850 raft_consensus.cc:374] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:34.713140 31850 raft_consensus.cc:729] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Initialized, Role: FOLLOWER
I20240503 08:11:34.713383 31850 consensus_queue.cc:260] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:34.713548 31850 raft_consensus.cc:388] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:34.713635 31850 raft_consensus.cc:482] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:34.713724 31850 raft_consensus.cc:3037] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:34.714197 31851 raft_consensus.cc:504] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:34.714491 31851 leader_election.cc:302] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81; no voters:
I20240503 08:11:34.714706 31851 leader_election.cc:288] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:34.714833 31841 raft_consensus.cc:2781] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:34.715611 31851 ts_tablet_manager.cc:1398] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:11:34.716797 31850 raft_consensus.cc:504] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:34.716822 31841 raft_consensus.cc:686] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Leader. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:34.717058 31850 leader_election.cc:302] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:34.717142 31841 consensus_queue.cc:237] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:34.717334 31837 raft_consensus.cc:2781] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:34.717519 31837 raft_consensus.cc:686] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Leader. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:34.717737 31837 consensus_queue.cc:237] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:34.718623 31850 leader_election.cc:288] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:34.719123 31850 ts_tablet_manager.cc:1398] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:11:34.720152 30663 catalog_manager.cc:5484] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193). New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:34.723307 30660 catalog_manager.cc:5484] T efe80a3ea5154c3a86d3816ba79657ce P ac3512789abc4bd890102fa6910f980b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3512789abc4bd890102fa6910f980b (127.27.157.194). New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } health_report { overall_health: HEALTHY } } }
W20240503 08:11:35.074004 31861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:35.074437 31861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:35.082798 31861 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 121.29 ms
time per row: 0.0606448 ms
I20240503 08:11:35.349781 30660 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56466:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:11:35.358450 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 38d8172f2dd1427fb7123752835bd458 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9a0826219eb944058c62c4f29729bf0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:35.358695 30921 tablet_service.cc:1467] Processing CreateTablet for tablet bee6e9222daa40e09dbf75c0ffaf479d (DEFAULT_TABLE table=rebalance_test_table_4 [id=9a0826219eb944058c62c4f29729bf0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:35.358961 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d8172f2dd1427fb7123752835bd458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:35.359141 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee6e9222daa40e09dbf75c0ffaf479d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:35.359166 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 35b0575c97db429fb6c30261552e090b (DEFAULT_TABLE table=rebalance_test_table_4 [id=9a0826219eb944058c62c4f29729bf0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:35.359606 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b0575c97db429fb6c30261552e090b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:35.365774 31903 tablet_bootstrap.cc:492] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: Bootstrap starting.
I20240503 08:11:35.366344 31904 tablet_bootstrap.cc:492] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: Bootstrap starting.
I20240503 08:11:35.367431 31903 tablet_bootstrap.cc:654] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:35.367727 31904 tablet_bootstrap.cc:654] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:35.369503 31903 tablet_bootstrap.cc:492] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: No bootstrap required, opened a new log
I20240503 08:11:35.369632 31904 tablet_bootstrap.cc:492] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: No bootstrap required, opened a new log
I20240503 08:11:35.369648 31903 ts_tablet_manager.cc:1367] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:11:35.369786 31904 ts_tablet_manager.cc:1367] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:35.370287 31903 raft_consensus.cc:348] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:35.370365 31904 raft_consensus.cc:348] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:35.370442 31903 raft_consensus.cc:374] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:35.370517 31904 raft_consensus.cc:374] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:35.370537 31903 raft_consensus.cc:729] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Initialized, Role: FOLLOWER
I20240503 08:11:35.370606 31904 raft_consensus.cc:729] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Initialized, Role: FOLLOWER
I20240503 08:11:35.370782 31903 consensus_queue.cc:260] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:35.370848 31904 consensus_queue.cc:260] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:35.370949 31903 raft_consensus.cc:388] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:35.371032 31903 raft_consensus.cc:482] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:35.371032 31904 raft_consensus.cc:388] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:35.371107 31904 raft_consensus.cc:482] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:35.371176 31903 raft_consensus.cc:3037] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:35.371213 31904 raft_consensus.cc:3037] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:35.373203 31903 raft_consensus.cc:504] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:35.373263 31904 raft_consensus.cc:504] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:35.373476 31903 leader_election.cc:302] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81; no voters:
I20240503 08:11:35.373495 31904 leader_election.cc:302] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:35.373734 31904 leader_election.cc:288] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:35.373791 31841 raft_consensus.cc:2781] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:35.373823 31837 raft_consensus.cc:2781] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:35.374328 31904 ts_tablet_manager.cc:1398] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:11:35.374753 31841 raft_consensus.cc:686] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Leader. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:35.374991 31841 consensus_queue.cc:237] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:35.375458 31903 leader_election.cc:288] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:35.376156 31903 ts_tablet_manager.cc:1398] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240503 08:11:35.376680 31903 tablet_bootstrap.cc:492] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: Bootstrap starting.
I20240503 08:11:35.380532 31903 tablet_bootstrap.cc:654] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:35.382171 31837 raft_consensus.cc:686] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Leader. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:35.382496 31837 consensus_queue.cc:237] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } }
I20240503 08:11:35.383554 31903 tablet_bootstrap.cc:492] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: No bootstrap required, opened a new log
I20240503 08:11:35.383736 31903 ts_tablet_manager.cc:1367] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240503 08:11:35.384440 31903 raft_consensus.cc:348] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:35.384687 31903 raft_consensus.cc:374] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:35.384771 31903 raft_consensus.cc:729] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Initialized, Role: FOLLOWER
I20240503 08:11:35.385998 30660 catalog_manager.cc:5484] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3512789abc4bd890102fa6910f980b (127.27.157.194). New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:35.386261 31903 consensus_queue.cc:260] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:35.386603 31903 raft_consensus.cc:388] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:35.386772 31903 raft_consensus.cc:482] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:35.386904 31903 raft_consensus.cc:3037] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:35.389429 31903 raft_consensus.cc:504] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:35.389763 31903 leader_election.cc:302] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81; no voters:
I20240503 08:11:35.390020 31903 leader_election.cc:288] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:35.390131 31907 raft_consensus.cc:2781] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:35.390909 31903 ts_tablet_manager.cc:1398] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240503 08:11:35.392581 31907 raft_consensus.cc:686] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Leader. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:35.395076 30666 catalog_manager.cc:5484] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193). New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:35.395625 31907 consensus_queue.cc:237] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } }
I20240503 08:11:35.401381 30666 catalog_manager.cc:5484] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193). New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } health_report { overall_health: HEALTHY } } }
W20240503 08:11:35.755028 31915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:35.755414 31915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:35.763068 31915 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 134.618 ms
time per row: 0.0673088 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:11:36.990096 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.195:42421
--local_ip_for_outbound_sockets=127.27.157.195
--tserver_master_addrs=127.27.157.254:36149
--webserver_port=36979
--webserver_interface=127.27.157.195
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:37.337111 31959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:37.337458 31959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:37.337608 31959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:37.345901 31959 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:37.346084 31959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:37.346278 31959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.195
I20240503 08:11:37.356215 31959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.195:42421
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.157.195
--webserver_port=36979
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.195
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:37.357627 31959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:37.359724 31959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:37.374917 31964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.375509 31965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.376376 31967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:37.377425 31959 server_base.cc:1034] running on GCE node
I20240503 08:11:37.378116 31959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:37.379261 31959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:37.380535 31959 hybrid_clock.cc:648] HybridClock initialized: now 1714723897380476 us; error 49 us; skew 500 ppm
I20240503 08:11:37.380964 31959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:37.383790 31959 webserver.cc:460] Webserver started at http://127.27.157.195:36979/ using document root <none> and password file <none>
I20240503 08:11:37.384678 31959 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:37.384816 31959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:37.390686 31959 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20240503 08:11:37.394002 31973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.395479 31959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:37.395660 31959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a9b83be26ed14471b0335ffb28f12193"
format_stamp: "Formatted at 2024-05-03 08:11:25 on dist-test-slave-dj8w"
I20240503 08:11:37.396550 31959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:37.416232 31959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:37.417105 31959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:37.417420 31959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:37.418454 31959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:37.420125 31959 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:37.420208 31959 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.420341 31959 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:37.420398 31959 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.454113 31959 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.195:42421
I20240503 08:11:37.454155 32085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.195:42421 every 8 connection(s)
I20240503 08:11:37.455662 31959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:37.460638 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 31959
I20240503 08:11:37.461230 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.196:42725
--local_ip_for_outbound_sockets=127.27.157.196
--tserver_master_addrs=127.27.157.254:36149
--webserver_port=46165
--webserver_interface=127.27.157.196
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:37.469704 32086 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:37.470078 32086 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:37.471192 32086 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:37.473104 30668 ts_manager.cc:194] Re-registered known tserver with Master: a9b83be26ed14471b0335ffb28f12193 (127.27.157.195:42421)
I20240503 08:11:37.473960 30668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.195:50403
W20240503 08:11:37.807935 32090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:37.808284 32090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:37.808420 32090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:37.816484 32090 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:37.816651 32090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:37.816812 32090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.196
I20240503 08:11:37.826467 32090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.196:42725
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.157.196
--webserver_port=46165
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.196
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:37.827980 32090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:37.829861 32090 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:37.843181 32095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.844135 32096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.845054 32098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:37.845846 32090 server_base.cc:1034] running on GCE node
I20240503 08:11:37.846506 32090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:37.847623 32090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:37.848906 32090 hybrid_clock.cc:648] HybridClock initialized: now 1714723897848830 us; error 76 us; skew 500 ppm
I20240503 08:11:37.849323 32090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:37.852195 32090 webserver.cc:460] Webserver started at http://127.27.157.196:46165/ using document root <none> and password file <none>
I20240503 08:11:37.853035 32090 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:37.853184 32090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:37.859218 32090 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 08:11:37.862556 32104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.864352 32090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 08:11:37.864562 32090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a7b227b516a24a1b837415111eba1303"
format_stamp: "Formatted at 2024-05-03 08:11:26 on dist-test-slave-dj8w"
I20240503 08:11:37.865532 32090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:37.887614 32090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:37.888567 32090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:37.888908 32090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:37.889953 32090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:37.891613 32090 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:37.891692 32090 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.891827 32090 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:37.891897 32090 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.926249 32090 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.196:42725
I20240503 08:11:37.926287 32216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.196:42725 every 8 connection(s)
I20240503 08:11:37.927855 32090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:37.931444 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 32090
I20240503 08:11:37.932267 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.197:34939
--local_ip_for_outbound_sockets=127.27.157.197
--tserver_master_addrs=127.27.157.254:36149
--webserver_port=38913
--webserver_interface=127.27.157.197
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:37.943935 32217 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:37.944268 32217 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:37.945175 32217 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:37.946753 30668 ts_manager.cc:194] Re-registered known tserver with Master: a7b227b516a24a1b837415111eba1303 (127.27.157.196:42725)
I20240503 08:11:37.947639 30668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.196:46857
W20240503 08:11:38.288584 32220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:38.288930 32220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:38.289088 32220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:38.297093 32220 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:38.297240 32220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:38.297369 32220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.197
I20240503 08:11:38.307063 32220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.197:34939
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.157.197
--webserver_port=38913
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.197
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:38.308512 32220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:38.310356 32220 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:38.323827 32226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:38.324702 32228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:38.325485 32225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:38.326561 32220 server_base.cc:1034] running on GCE node
I20240503 08:11:38.327230 32220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:38.328370 32220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:38.329587 32220 hybrid_clock.cc:648] HybridClock initialized: now 1714723898329526 us; error 51 us; skew 500 ppm
I20240503 08:11:38.329986 32220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:38.337529 32220 webserver.cc:460] Webserver started at http://127.27.157.197:38913/ using document root <none> and password file <none>
I20240503 08:11:38.338378 32220 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:38.338527 32220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:38.344434 32220 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:11:38.347674 32234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.349684 32220 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:11:38.349897 32220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c5dc2c8f44d44889b3888f6265d07de9"
format_stamp: "Formatted at 2024-05-03 08:11:26 on dist-test-slave-dj8w"
I20240503 08:11:38.350869 32220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:38.371201 32220 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:38.372196 32220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:38.372488 32220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:38.373637 32220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:38.375838 32241 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 08:11:38.381494 32220 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:11:38.381649 32220 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20240503 08:11:38.381835 32220 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:11:38.385159 32220 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:11:38.385286 32220 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:38.385813 32241 tablet_bootstrap.cc:492] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: Bootstrap starting.
I20240503 08:11:38.407034 32241 log.cc:824] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:38.412202 32241 tablet_bootstrap.cc:492] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:38.412992 32241 tablet_bootstrap.cc:492] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: Bootstrap complete.
I20240503 08:11:38.414180 32241 ts_tablet_manager.cc:1367] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.016s
I20240503 08:11:38.425951 32241 raft_consensus.cc:348] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } }
I20240503 08:11:38.426826 32241 raft_consensus.cc:729] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5dc2c8f44d44889b3888f6265d07de9, State: Initialized, Role: FOLLOWER
I20240503 08:11:38.427742 32241 consensus_queue.cc:260] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } }
I20240503 08:11:38.428037 32241 raft_consensus.cc:388] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:38.428236 32241 raft_consensus.cc:482] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:38.428445 32241 raft_consensus.cc:3037] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:38.433442 32241 raft_consensus.cc:504] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } }
I20240503 08:11:38.434244 32241 leader_election.cc:302] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5dc2c8f44d44889b3888f6265d07de9; no voters:
I20240503 08:11:38.435206 32241 leader_election.cc:288] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 08:11:38.435390 32325 raft_consensus.cc:2781] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:38.437788 32241 ts_tablet_manager.cc:1398] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9: Time spent starting tablet: real 0.023s user 0.019s sys 0.005s
I20240503 08:11:38.438524 32325 raft_consensus.cc:686] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [term 2 LEADER]: Becoming Leader. State: Replica: c5dc2c8f44d44889b3888f6265d07de9, State: Running, Role: LEADER
I20240503 08:11:38.439169 32325 consensus_queue.cc:237] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } }
I20240503 08:11:38.448810 32220 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.197:34939
I20240503 08:11:38.448868 32353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.197:34939 every 8 connection(s)
I20240503 08:11:38.450229 32220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:38.456588 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 32220
I20240503 08:11:38.457237 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.198:42339
--local_ip_for_outbound_sockets=127.27.157.198
--tserver_master_addrs=127.27.157.254:36149
--webserver_port=46543
--webserver_interface=127.27.157.198
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:38.464268 32354 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:38.464607 32354 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:38.465476 32354 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:38.468293 30668 ts_manager.cc:194] Re-registered known tserver with Master: c5dc2c8f44d44889b3888f6265d07de9 (127.27.157.197:34939)
I20240503 08:11:38.468698 30668 catalog_manager.cc:5484] T ded8fc9e27bf4613bf2f3dd4e9db250a P c5dc2c8f44d44889b3888f6265d07de9 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c5dc2c8f44d44889b3888f6265d07de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:38.473819 30668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.197:46821
I20240503 08:11:38.476652 32354 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
I20240503 08:11:38.477491 32086 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
W20240503 08:11:38.807118 32358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:38.807476 32358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:38.807621 32358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:38.815443 32358 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:38.815589 32358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:38.815768 32358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.198
I20240503 08:11:38.825350 32358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.198:42339
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.157.198
--webserver_port=46543
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.198
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:38.826923 32358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:38.828860 32358 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:38.842067 32367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:38.842967 32368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:38.843686 32358 server_base.cc:1034] running on GCE node
W20240503 08:11:38.844626 32370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:38.845364 32358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:38.846387 32358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:38.847611 32358 hybrid_clock.cc:648] HybridClock initialized: now 1714723898847554 us; error 44 us; skew 500 ppm
I20240503 08:11:38.847982 32358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:38.850816 32358 webserver.cc:460] Webserver started at http://127.27.157.198:46543/ using document root <none> and password file <none>
I20240503 08:11:38.851922 32358 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:38.852072 32358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:38.858188 32358 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:11:38.861431 32376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.863029 32358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:38.863221 32358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9f45625e8fff471382d1376ff8cf0d00"
format_stamp: "Formatted at 2024-05-03 08:11:27 on dist-test-slave-dj8w"
I20240503 08:11:38.864207 32358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:38.880851 32358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:38.881743 32358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:38.882071 32358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:38.883103 32358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:38.884908 32358 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:38.884996 32358 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.885135 32358 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:38.885192 32358 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:38.921751 32358 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.198:42339
I20240503 08:11:38.921770 32488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.198:42339 every 8 connection(s)
I20240503 08:11:38.923497 32358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:11:38.928797 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 32358
I20240503 08:11:38.929633 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.199:36035
--local_ip_for_outbound_sockets=127.27.157.199
--tserver_master_addrs=127.27.157.254:36149
--webserver_port=33489
--webserver_interface=127.27.157.199
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:38.939121 32489 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:38.939551 32489 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:38.940704 32489 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:38.942545 30668 ts_manager.cc:194] Re-registered known tserver with Master: 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198:42339)
I20240503 08:11:38.943589 30668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.198:37183
I20240503 08:11:38.950285 32217 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
W20240503 08:11:39.293331 32492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:39.293696 32492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:39.293859 32492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:39.302227 32492 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:39.302390 32492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:39.302553 32492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.199
I20240503 08:11:39.312614 32492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.199:36035
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.157.199
--webserver_port=33489
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.199
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:39.314079 32492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:39.316124 32492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:39.332494 32498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:39.332928 32497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:39.333853 32500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:39.334965 32492 server_base.cc:1034] running on GCE node
I20240503 08:11:39.335670 32492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:39.336788 32492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:39.338024 32492 hybrid_clock.cc:648] HybridClock initialized: now 1714723899337973 us; error 45 us; skew 500 ppm
I20240503 08:11:39.338371 32492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:39.341010 32492 webserver.cc:460] Webserver started at http://127.27.157.199:33489/ using document root <none> and password file <none>
I20240503 08:11:39.341843 32492 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:39.341987 32492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:39.347936 32492 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:11:39.351116 32506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:39.353504 32492 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:11:39.353700 32492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "03e83fbce8ee4c5cacbf54f25e1f2665"
format_stamp: "Formatted at 2024-05-03 08:11:27 on dist-test-slave-dj8w"
I20240503 08:11:39.354605 32492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:39.378456 32492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:39.379581 32492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:39.379902 32492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:39.381017 32492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:39.382745 32492 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:39.382833 32492 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:39.382966 32492 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:39.383024 32492 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:39.432462 32492 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.199:36035
I20240503 08:11:39.432528 32618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.199:36035 every 8 connection(s)
I20240503 08:11:39.434243 32492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:11:39.444226 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 32492
I20240503 08:11:39.453522 32619 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:39.453922 32619 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:39.455145 32619 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:39.457084 30668 ts_manager.cc:194] Re-registered known tserver with Master: 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199:36035)
I20240503 08:11:39.458410 30668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.199:49765
I20240503 08:11:39.947156 32489 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
I20240503 08:11:39.982751 32424 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:39.988270 32554 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:39.992870 32021 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:39.993197 32152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.001528 30921 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:40.006167 32283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:40.015620 30790 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:40.236096 30790 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:11:40.255993 32554 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.257297 32152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.258739 32021 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.264943 32424 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:40.281225 30921 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:11:40.298491 32283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:11:40.381194 30810 consensus_queue.cc:237] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: true } }
I20240503 08:11:40.383983 32711 raft_consensus.cc:2932] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a7b227b516a24a1b837415111eba1303 (127.27.157.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: true } } }
I20240503 08:11:40.390342 30667 catalog_manager.cc:5484] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index -1 to 4, NON_VOTER a7b227b516a24a1b837415111eba1303 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:40.393186 30744 consensus_peers.cc:485] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 -> Peer a7b227b516a24a1b837415111eba1303 (127.27.157.196:42725): Couldn't send request to peer a7b227b516a24a1b837415111eba1303. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35b0575c97db429fb6c30261552e090b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:40.407618 30941 consensus_queue.cc:237] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: true } }
I20240503 08:11:40.410485 32724 raft_consensus.cc:2932] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: true } } }
I20240503 08:11:40.418758 30660 catalog_manager.cc:5484] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b reported cstate change: config changed from index -1 to 4, NON_VOTER 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:40.422295 30876 consensus_peers.cc:485] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b -> Peer 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198:42339): Couldn't send request to peer 9f45625e8fff471382d1376ff8cf0d00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa28377977cf4ffa9347b5d45020c377. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:40.433367 30810 consensus_queue.cc:237] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: true } }
I20240503 08:11:40.435559 32711 raft_consensus.cc:2932] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a9b83be26ed14471b0335ffb28f12193 (127.27.157.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: true } } }
I20240503 08:11:40.441751 30660 catalog_manager.cc:5484] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index -1 to 4, NON_VOTER a9b83be26ed14471b0335ffb28f12193 (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:40.446283 30746 consensus_peers.cc:485] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 -> Peer a9b83be26ed14471b0335ffb28f12193 (127.27.157.195:42421): Couldn't send request to peer a9b83be26ed14471b0335ffb28f12193. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b123f5c42af4d11b4325b3cd6226c29. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:40.458496 30941 consensus_queue.cc:237] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: true } }
I20240503 08:11:40.460594 32724 raft_consensus.cc:2932] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: true } } }
I20240503 08:11:40.462271 32619 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
I20240503 08:11:40.466992 30660 catalog_manager.cc:5484] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b reported cstate change: config changed from index -1 to 4, NON_VOTER 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:40.469475 30874 consensus_peers.cc:485] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b -> Peer 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199:36035): Couldn't send request to peer 03e83fbce8ee4c5cacbf54f25e1f2665. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20b5e4e7b78f4f0d979b936d92dd1eb3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:40.480182 30810 consensus_queue.cc:237] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: true } }
I20240503 08:11:40.482589 32711 raft_consensus.cc:2932] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5dc2c8f44d44889b3888f6265d07de9 (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: true } } }
I20240503 08:11:40.489264 30660 catalog_manager.cc:5484] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index -1 to 4, NON_VOTER c5dc2c8f44d44889b3888f6265d07de9 (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:40.491044 30743 consensus_peers.cc:485] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 -> Peer c5dc2c8f44d44889b3888f6265d07de9 (127.27.157.197:34939): Couldn't send request to peer c5dc2c8f44d44889b3888f6265d07de9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f4d7f23456f4bddabcdd74ddea40a33. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:40.512750 30810 consensus_queue.cc:237] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: true } }
I20240503 08:11:40.514672 32711 raft_consensus.cc:2932] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a7b227b516a24a1b837415111eba1303 (127.27.157.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: true } } }
W20240503 08:11:40.515288 30744 consensus_peers.cc:485] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 -> Peer a7b227b516a24a1b837415111eba1303 (127.27.157.196:42725): Couldn't send request to peer a7b227b516a24a1b837415111eba1303. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 153c5b78de3342eab81fdce1a7f199d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:40.519290 30660 catalog_manager.cc:5484] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index -1 to 4, NON_VOTER a7b227b516a24a1b837415111eba1303 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:40.532646 30941 consensus_queue.cc:237] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: true } }
I20240503 08:11:40.534500 32724 raft_consensus.cc:2932] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: true } } }
W20240503 08:11:40.535112 30876 consensus_peers.cc:485] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b -> Peer 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198:42339): Couldn't send request to peer 9f45625e8fff471382d1376ff8cf0d00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bee6e9222daa40e09dbf75c0ffaf479d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:40.539479 30660 catalog_manager.cc:5484] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b reported cstate change: config changed from index -1 to 4, NON_VOTER 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:40.550470 30941 consensus_queue.cc:237] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: true } }
I20240503 08:11:40.552419 32756 raft_consensus.cc:2932] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a9b83be26ed14471b0335ffb28f12193 (127.27.157.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: true } } }
I20240503 08:11:40.558274 30660 catalog_manager.cc:5484] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b reported cstate change: config changed from index -1 to 4, NON_VOTER a9b83be26ed14471b0335ffb28f12193 (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:40.559398 30877 consensus_peers.cc:485] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b -> Peer a9b83be26ed14471b0335ffb28f12193 (127.27.157.195:42421): Couldn't send request to peer a9b83be26ed14471b0335ffb28f12193. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e188dd3547440b183799c9a796b4a85. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:40.570488 30810 consensus_queue.cc:237] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: true } }
I20240503 08:11:40.572391 32710 raft_consensus.cc:2932] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: true } } }
I20240503 08:11:40.578632 30660 catalog_manager.cc:5484] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index -1 to 4, NON_VOTER 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:40.580847 30743 consensus_peers.cc:485] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 -> Peer 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199:36035): Couldn't send request to peer 03e83fbce8ee4c5cacbf54f25e1f2665. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b5f4bb2e73c4643a964b3c834b820aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:40.769706 346 ts_tablet_manager.cc:902] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303: Initiating tablet copy from peer 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:40.771639 346 tablet_copy_client.cc:308] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:44943
I20240503 08:11:40.778998 30830 tablet_copy_service.cc:140] P 3986faa56d004ea5b787a469ddb79c81: Received BeginTabletCopySession request for tablet 35b0575c97db429fb6c30261552e090b from peer a7b227b516a24a1b837415111eba1303 ({username='slave'} at 127.27.157.196:36101)
I20240503 08:11:40.779388 30830 tablet_copy_service.cc:161] P 3986faa56d004ea5b787a469ddb79c81: Beginning new tablet copy session on tablet 35b0575c97db429fb6c30261552e090b from peer a7b227b516a24a1b837415111eba1303 at {username='slave'} at 127.27.157.196:36101: session id = a7b227b516a24a1b837415111eba1303-35b0575c97db429fb6c30261552e090b
I20240503 08:11:40.782231 30830 tablet_copy_source_session.cc:205] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:40.787904 346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b0575c97db429fb6c30261552e090b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:40.799170 346 tablet_copy_client.cc:783] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:40.799618 346 tablet_copy_client.cc:647] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:40.802493 346 tablet_copy_client.cc:523] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:40.806068 346 tablet_bootstrap.cc:492] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303: Bootstrap starting.
I20240503 08:11:40.816690 346 log.cc:824] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:40.849918 346 tablet_bootstrap.cc:492] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:40.850464 346 tablet_bootstrap.cc:492] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303: Bootstrap complete.
I20240503 08:11:40.850917 346 ts_tablet_manager.cc:1367] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303: Time spent bootstrapping tablet: real 0.045s user 0.039s sys 0.005s
I20240503 08:11:40.855648 346 raft_consensus.cc:348] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: true } }
I20240503 08:11:40.856106 346 raft_consensus.cc:729] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7b227b516a24a1b837415111eba1303, State: Initialized, Role: LEARNER
I20240503 08:11:40.856604 346 consensus_queue.cc:260] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: true } }
I20240503 08:11:40.858433 346 ts_tablet_manager.cc:1398] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303: Time spent starting tablet: real 0.007s user 0.010s sys 0.000s
I20240503 08:11:40.859673 30830 tablet_copy_service.cc:342] P 3986faa56d004ea5b787a469ddb79c81: Request end of tablet copy session a7b227b516a24a1b837415111eba1303-35b0575c97db429fb6c30261552e090b received from {username='slave'} at 127.27.157.196:36101
I20240503 08:11:40.859895 30830 tablet_copy_service.cc:433] P 3986faa56d004ea5b787a469ddb79c81: ending tablet copy session a7b227b516a24a1b837415111eba1303-35b0575c97db429fb6c30261552e090b on tablet 35b0575c97db429fb6c30261552e090b with peer a7b227b516a24a1b837415111eba1303
I20240503 08:11:40.907433 346 ts_tablet_manager.cc:902] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303: Initiating tablet copy from peer 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:40.909811 346 tablet_copy_client.cc:308] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:44943
I20240503 08:11:40.910820 30830 tablet_copy_service.cc:140] P 3986faa56d004ea5b787a469ddb79c81: Received BeginTabletCopySession request for tablet 153c5b78de3342eab81fdce1a7f199d5 from peer a7b227b516a24a1b837415111eba1303 ({username='slave'} at 127.27.157.196:36101)
I20240503 08:11:40.911018 30830 tablet_copy_service.cc:161] P 3986faa56d004ea5b787a469ddb79c81: Beginning new tablet copy session on tablet 153c5b78de3342eab81fdce1a7f199d5 from peer a7b227b516a24a1b837415111eba1303 at {username='slave'} at 127.27.157.196:36101: session id = a7b227b516a24a1b837415111eba1303-153c5b78de3342eab81fdce1a7f199d5
I20240503 08:11:40.913359 30830 tablet_copy_source_session.cc:205] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:40.914664 346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153c5b78de3342eab81fdce1a7f199d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:40.922026 346 tablet_copy_client.cc:783] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:40.922420 346 tablet_copy_client.cc:647] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:40.924860 346 tablet_copy_client.cc:523] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:40.936800 346 tablet_bootstrap.cc:492] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303: Bootstrap starting.
I20240503 08:11:40.939849 377 ts_tablet_manager.cc:902] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193: Initiating tablet copy from peer ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429)
I20240503 08:11:40.956331 377 tablet_copy_client.cc:308] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:36429
I20240503 08:11:40.980618 386 ts_tablet_manager.cc:902] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00: Initiating tablet copy from peer ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429)
I20240503 08:11:40.983353 30961 tablet_copy_service.cc:140] P ac3512789abc4bd890102fa6910f980b: Received BeginTabletCopySession request for tablet 6e188dd3547440b183799c9a796b4a85 from peer a9b83be26ed14471b0335ffb28f12193 ({username='slave'} at 127.27.157.195:39687)
I20240503 08:11:40.983731 30961 tablet_copy_service.cc:161] P ac3512789abc4bd890102fa6910f980b: Beginning new tablet copy session on tablet 6e188dd3547440b183799c9a796b4a85 from peer a9b83be26ed14471b0335ffb28f12193 at {username='slave'} at 127.27.157.195:39687: session id = a9b83be26ed14471b0335ffb28f12193-6e188dd3547440b183799c9a796b4a85
I20240503 08:11:40.984894 387 ts_tablet_manager.cc:902] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665: Initiating tablet copy from peer ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429)
I20240503 08:11:40.986990 30961 tablet_copy_source_session.cc:205] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:40.995292 377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e188dd3547440b183799c9a796b4a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:40.998167 397 ts_tablet_manager.cc:902] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00: Initiating tablet copy from peer ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429)
I20240503 08:11:41.019120 386 tablet_copy_client.cc:308] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:36429
I20240503 08:11:41.021732 377 tablet_copy_client.cc:783] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:41.021891 387 tablet_copy_client.cc:308] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:36429
I20240503 08:11:41.022387 377 tablet_copy_client.cc:647] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:41.027793 377 tablet_copy_client.cc:523] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:41.039093 377 tablet_bootstrap.cc:492] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193: Bootstrap starting.
I20240503 08:11:41.050172 30961 tablet_copy_service.cc:140] P ac3512789abc4bd890102fa6910f980b: Received BeginTabletCopySession request for tablet 20b5e4e7b78f4f0d979b936d92dd1eb3 from peer 03e83fbce8ee4c5cacbf54f25e1f2665 ({username='slave'} at 127.27.157.199:40367)
I20240503 08:11:41.050407 30961 tablet_copy_service.cc:161] P ac3512789abc4bd890102fa6910f980b: Beginning new tablet copy session on tablet 20b5e4e7b78f4f0d979b936d92dd1eb3 from peer 03e83fbce8ee4c5cacbf54f25e1f2665 at {username='slave'} at 127.27.157.199:40367: session id = 03e83fbce8ee4c5cacbf54f25e1f2665-20b5e4e7b78f4f0d979b936d92dd1eb3
I20240503 08:11:41.054986 30961 tablet_copy_source_session.cc:205] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:41.061100 397 tablet_copy_client.cc:308] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:36429
I20240503 08:11:41.061424 387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b5e4e7b78f4f0d979b936d92dd1eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:41.059269 414 ts_tablet_manager.cc:902] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9: Initiating tablet copy from peer 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:41.065474 30961 tablet_copy_service.cc:140] P ac3512789abc4bd890102fa6910f980b: Received BeginTabletCopySession request for tablet bee6e9222daa40e09dbf75c0ffaf479d from peer 9f45625e8fff471382d1376ff8cf0d00 ({username='slave'} at 127.27.157.198:39269)
I20240503 08:11:41.065690 30961 tablet_copy_service.cc:161] P ac3512789abc4bd890102fa6910f980b: Beginning new tablet copy session on tablet bee6e9222daa40e09dbf75c0ffaf479d from peer 9f45625e8fff471382d1376ff8cf0d00 at {username='slave'} at 127.27.157.198:39269: session id = 9f45625e8fff471382d1376ff8cf0d00-bee6e9222daa40e09dbf75c0ffaf479d
I20240503 08:11:41.066321 377 log.cc:824] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:41.069641 30961 tablet_copy_source_session.cc:205] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:41.071784 30961 tablet_copy_service.cc:140] P ac3512789abc4bd890102fa6910f980b: Received BeginTabletCopySession request for tablet fa28377977cf4ffa9347b5d45020c377 from peer 9f45625e8fff471382d1376ff8cf0d00 ({username='slave'} at 127.27.157.198:39269)
I20240503 08:11:41.072041 30961 tablet_copy_service.cc:161] P ac3512789abc4bd890102fa6910f980b: Beginning new tablet copy session on tablet fa28377977cf4ffa9347b5d45020c377 from peer 9f45625e8fff471382d1376ff8cf0d00 at {username='slave'} at 127.27.157.198:39269: session id = 9f45625e8fff471382d1376ff8cf0d00-fa28377977cf4ffa9347b5d45020c377
I20240503 08:11:41.082221 386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee6e9222daa40e09dbf75c0ffaf479d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:41.084957 346 tablet_bootstrap.cc:492] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:41.085119 30961 tablet_copy_source_session.cc:205] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:41.085522 346 tablet_bootstrap.cc:492] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303: Bootstrap complete.
I20240503 08:11:41.085906 346 ts_tablet_manager.cc:1367] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303: Time spent bootstrapping tablet: real 0.149s user 0.069s sys 0.008s
I20240503 08:11:41.087227 346 raft_consensus.cc:348] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: true } }
I20240503 08:11:41.087587 346 raft_consensus.cc:729] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7b227b516a24a1b837415111eba1303, State: Initialized, Role: LEARNER
I20240503 08:11:41.087858 346 consensus_queue.cc:260] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: true } }
I20240503 08:11:41.092018 397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa28377977cf4ffa9347b5d45020c377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:41.092746 412 ts_tablet_manager.cc:902] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193: Initiating tablet copy from peer 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:41.094520 346 ts_tablet_manager.cc:1398] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 08:11:41.095485 30830 tablet_copy_service.cc:342] P 3986faa56d004ea5b787a469ddb79c81: Request end of tablet copy session a7b227b516a24a1b837415111eba1303-153c5b78de3342eab81fdce1a7f199d5 received from {username='slave'} at 127.27.157.196:36101
I20240503 08:11:41.095639 30830 tablet_copy_service.cc:433] P 3986faa56d004ea5b787a469ddb79c81: ending tablet copy session a7b227b516a24a1b837415111eba1303-153c5b78de3342eab81fdce1a7f199d5 on tablet 153c5b78de3342eab81fdce1a7f199d5 with peer a7b227b516a24a1b837415111eba1303
I20240503 08:11:41.099948 387 tablet_copy_client.cc:783] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:41.100421 387 tablet_copy_client.cc:647] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:41.103660 387 tablet_copy_client.cc:523] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:41.107877 414 tablet_copy_client.cc:308] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:44943
I20240503 08:11:41.109658 387 tablet_bootstrap.cc:492] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665: Bootstrap starting.
I20240503 08:11:41.114185 397 tablet_copy_client.cc:783] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:41.114985 397 tablet_copy_client.cc:647] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:41.118474 397 tablet_copy_client.cc:523] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:41.125932 386 tablet_copy_client.cc:783] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:41.126226 386 tablet_copy_client.cc:647] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:41.127475 397 tablet_bootstrap.cc:492] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00: Bootstrap starting.
I20240503 08:11:41.129884 386 tablet_copy_client.cc:523] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:41.135264 30830 tablet_copy_service.cc:140] P 3986faa56d004ea5b787a469ddb79c81: Received BeginTabletCopySession request for tablet 1f4d7f23456f4bddabcdd74ddea40a33 from peer c5dc2c8f44d44889b3888f6265d07de9 ({username='slave'} at 127.27.157.197:54591)
I20240503 08:11:41.135493 30830 tablet_copy_service.cc:161] P 3986faa56d004ea5b787a469ddb79c81: Beginning new tablet copy session on tablet 1f4d7f23456f4bddabcdd74ddea40a33 from peer c5dc2c8f44d44889b3888f6265d07de9 at {username='slave'} at 127.27.157.197:54591: session id = c5dc2c8f44d44889b3888f6265d07de9-1f4d7f23456f4bddabcdd74ddea40a33
I20240503 08:11:41.135941 387 log.cc:824] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:41.139554 30830 tablet_copy_source_session.cc:205] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:41.143477 414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4d7f23456f4bddabcdd74ddea40a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:41.144423 386 tablet_bootstrap.cc:492] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00: Bootstrap starting.
I20240503 08:11:41.161393 412 tablet_copy_client.cc:308] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:44943
I20240503 08:11:41.163100 414 tablet_copy_client.cc:783] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:41.163607 414 tablet_copy_client.cc:647] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:41.167982 414 tablet_copy_client.cc:523] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:41.172427 30830 tablet_copy_service.cc:140] P 3986faa56d004ea5b787a469ddb79c81: Received BeginTabletCopySession request for tablet 6b123f5c42af4d11b4325b3cd6226c29 from peer a9b83be26ed14471b0335ffb28f12193 ({username='slave'} at 127.27.157.195:35163)
I20240503 08:11:41.172655 30830 tablet_copy_service.cc:161] P 3986faa56d004ea5b787a469ddb79c81: Beginning new tablet copy session on tablet 6b123f5c42af4d11b4325b3cd6226c29 from peer a9b83be26ed14471b0335ffb28f12193 at {username='slave'} at 127.27.157.195:35163: session id = a9b83be26ed14471b0335ffb28f12193-6b123f5c42af4d11b4325b3cd6226c29
I20240503 08:11:41.175910 30830 tablet_copy_source_session.cc:205] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:41.178149 412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b123f5c42af4d11b4325b3cd6226c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:41.178913 430 ts_tablet_manager.cc:902] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665: Initiating tablet copy from peer 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:41.181048 414 tablet_bootstrap.cc:492] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9: Bootstrap starting.
I20240503 08:11:41.196211 397 log.cc:824] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:41.204618 412 tablet_copy_client.cc:783] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:41.205094 412 tablet_copy_client.cc:647] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:41.205220 430 tablet_copy_client.cc:308] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:44943
I20240503 08:11:41.207747 377 tablet_bootstrap.cc:492] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:41.208374 377 tablet_bootstrap.cc:492] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193: Bootstrap complete.
I20240503 08:11:41.208901 377 ts_tablet_manager.cc:1367] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193: Time spent bootstrapping tablet: real 0.170s user 0.072s sys 0.012s
I20240503 08:11:41.214756 30830 tablet_copy_service.cc:140] P 3986faa56d004ea5b787a469ddb79c81: Received BeginTabletCopySession request for tablet 2b5f4bb2e73c4643a964b3c834b820aa from peer 03e83fbce8ee4c5cacbf54f25e1f2665 ({username='slave'} at 127.27.157.199:43345)
I20240503 08:11:41.214949 30830 tablet_copy_service.cc:161] P 3986faa56d004ea5b787a469ddb79c81: Beginning new tablet copy session on tablet 2b5f4bb2e73c4643a964b3c834b820aa from peer 03e83fbce8ee4c5cacbf54f25e1f2665 at {username='slave'} at 127.27.157.199:43345: session id = 03e83fbce8ee4c5cacbf54f25e1f2665-2b5f4bb2e73c4643a964b3c834b820aa
I20240503 08:11:41.216722 412 tablet_copy_client.cc:523] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:41.218232 30830 tablet_copy_source_session.cc:205] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:41.219995 430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5f4bb2e73c4643a964b3c834b820aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:41.225172 412 tablet_bootstrap.cc:492] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193: Bootstrap starting.
I20240503 08:11:41.228912 430 tablet_copy_client.cc:783] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:41.229317 430 tablet_copy_client.cc:647] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:41.234145 430 tablet_copy_client.cc:523] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:41.237175 377 raft_consensus.cc:348] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: true } }
I20240503 08:11:41.237728 377 raft_consensus.cc:729] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9b83be26ed14471b0335ffb28f12193, State: Initialized, Role: LEARNER
I20240503 08:11:41.238353 377 consensus_queue.cc:260] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: true } }
I20240503 08:11:41.242555 430 tablet_bootstrap.cc:492] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665: Bootstrap starting.
I20240503 08:11:41.248831 377 ts_tablet_manager.cc:1398] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193: Time spent starting tablet: real 0.040s user 0.011s sys 0.000s
I20240503 08:11:41.251399 387 tablet_bootstrap.cc:492] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:41.252038 387 tablet_bootstrap.cc:492] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665: Bootstrap complete.
I20240503 08:11:41.252547 387 ts_tablet_manager.cc:1367] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665: Time spent bootstrapping tablet: real 0.143s user 0.089s sys 0.030s
I20240503 08:11:41.257609 30961 tablet_copy_service.cc:342] P ac3512789abc4bd890102fa6910f980b: Request end of tablet copy session a9b83be26ed14471b0335ffb28f12193-6e188dd3547440b183799c9a796b4a85 received from {username='slave'} at 127.27.157.195:39687
I20240503 08:11:41.257817 30961 tablet_copy_service.cc:433] P ac3512789abc4bd890102fa6910f980b: ending tablet copy session a9b83be26ed14471b0335ffb28f12193-6e188dd3547440b183799c9a796b4a85 on tablet 6e188dd3547440b183799c9a796b4a85 with peer a9b83be26ed14471b0335ffb28f12193
I20240503 08:11:41.259815 387 raft_consensus.cc:348] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: true } }
I20240503 08:11:41.260468 387 raft_consensus.cc:729] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03e83fbce8ee4c5cacbf54f25e1f2665, State: Initialized, Role: LEARNER
I20240503 08:11:41.261288 387 consensus_queue.cc:260] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: true } }
I20240503 08:11:41.270216 387 ts_tablet_manager.cc:1398] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
I20240503 08:11:41.274606 30961 tablet_copy_service.cc:342] P ac3512789abc4bd890102fa6910f980b: Request end of tablet copy session 03e83fbce8ee4c5cacbf54f25e1f2665-20b5e4e7b78f4f0d979b936d92dd1eb3 received from {username='slave'} at 127.27.157.199:40367
I20240503 08:11:41.274791 30961 tablet_copy_service.cc:433] P ac3512789abc4bd890102fa6910f980b: ending tablet copy session 03e83fbce8ee4c5cacbf54f25e1f2665-20b5e4e7b78f4f0d979b936d92dd1eb3 on tablet 20b5e4e7b78f4f0d979b936d92dd1eb3 with peer 03e83fbce8ee4c5cacbf54f25e1f2665
I20240503 08:11:41.276551 32172 raft_consensus.cc:1203] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:41.293198 414 tablet_bootstrap.cc:492] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:41.293804 414 tablet_bootstrap.cc:492] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9: Bootstrap complete.
I20240503 08:11:41.294178 414 ts_tablet_manager.cc:1367] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9: Time spent bootstrapping tablet: real 0.113s user 0.045s sys 0.012s
I20240503 08:11:41.295521 414 raft_consensus.cc:348] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: true } }
I20240503 08:11:41.295848 414 raft_consensus.cc:729] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5dc2c8f44d44889b3888f6265d07de9, State: Initialized, Role: LEARNER
I20240503 08:11:41.296152 414 consensus_queue.cc:260] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: true } }
I20240503 08:11:41.304186 414 ts_tablet_manager.cc:1398] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240503 08:11:41.305833 30830 tablet_copy_service.cc:342] P 3986faa56d004ea5b787a469ddb79c81: Request end of tablet copy session c5dc2c8f44d44889b3888f6265d07de9-1f4d7f23456f4bddabcdd74ddea40a33 received from {username='slave'} at 127.27.157.197:54591
I20240503 08:11:41.306035 30830 tablet_copy_service.cc:433] P 3986faa56d004ea5b787a469ddb79c81: ending tablet copy session c5dc2c8f44d44889b3888f6265d07de9-1f4d7f23456f4bddabcdd74ddea40a33 on tablet 1f4d7f23456f4bddabcdd74ddea40a33 with peer c5dc2c8f44d44889b3888f6265d07de9
I20240503 08:11:41.320318 430 tablet_bootstrap.cc:492] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:41.320824 430 tablet_bootstrap.cc:492] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665: Bootstrap complete.
I20240503 08:11:41.321141 430 ts_tablet_manager.cc:1367] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665: Time spent bootstrapping tablet: real 0.079s user 0.054s sys 0.004s
I20240503 08:11:41.322345 430 raft_consensus.cc:348] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: true } }
I20240503 08:11:41.322628 430 raft_consensus.cc:729] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03e83fbce8ee4c5cacbf54f25e1f2665, State: Initialized, Role: LEARNER
I20240503 08:11:41.322839 430 consensus_queue.cc:260] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: true } }
I20240503 08:11:41.326558 430 ts_tablet_manager.cc:1398] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:11:41.329102 386 tablet_bootstrap.cc:492] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:41.329869 386 tablet_bootstrap.cc:492] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00: Bootstrap complete.
I20240503 08:11:41.330045 30830 tablet_copy_service.cc:342] P 3986faa56d004ea5b787a469ddb79c81: Request end of tablet copy session 03e83fbce8ee4c5cacbf54f25e1f2665-2b5f4bb2e73c4643a964b3c834b820aa received from {username='slave'} at 127.27.157.199:43345
I20240503 08:11:41.330214 30830 tablet_copy_service.cc:433] P 3986faa56d004ea5b787a469ddb79c81: ending tablet copy session 03e83fbce8ee4c5cacbf54f25e1f2665-2b5f4bb2e73c4643a964b3c834b820aa on tablet 2b5f4bb2e73c4643a964b3c834b820aa with peer 03e83fbce8ee4c5cacbf54f25e1f2665
I20240503 08:11:41.330482 386 ts_tablet_manager.cc:1367] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00: Time spent bootstrapping tablet: real 0.186s user 0.056s sys 0.012s
I20240503 08:11:41.332599 412 tablet_bootstrap.cc:492] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:41.333002 412 tablet_bootstrap.cc:492] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193: Bootstrap complete.
I20240503 08:11:41.333273 412 ts_tablet_manager.cc:1367] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193: Time spent bootstrapping tablet: real 0.108s user 0.047s sys 0.008s
I20240503 08:11:41.334100 412 raft_consensus.cc:348] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: true } }
I20240503 08:11:41.334326 412 raft_consensus.cc:729] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9b83be26ed14471b0335ffb28f12193, State: Initialized, Role: LEARNER
I20240503 08:11:41.334556 412 consensus_queue.cc:260] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: true } }
I20240503 08:11:41.335196 412 ts_tablet_manager.cc:1398] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:11:41.336189 30830 tablet_copy_service.cc:342] P 3986faa56d004ea5b787a469ddb79c81: Request end of tablet copy session a9b83be26ed14471b0335ffb28f12193-6b123f5c42af4d11b4325b3cd6226c29 received from {username='slave'} at 127.27.157.195:35163
I20240503 08:11:41.336365 30830 tablet_copy_service.cc:433] P 3986faa56d004ea5b787a469ddb79c81: ending tablet copy session a9b83be26ed14471b0335ffb28f12193-6b123f5c42af4d11b4325b3cd6226c29 on tablet 6b123f5c42af4d11b4325b3cd6226c29 with peer a9b83be26ed14471b0335ffb28f12193
I20240503 08:11:41.337895 386 raft_consensus.cc:348] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: true } }
I20240503 08:11:41.338483 386 raft_consensus.cc:729] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f45625e8fff471382d1376ff8cf0d00, State: Initialized, Role: LEARNER
I20240503 08:11:41.339143 386 consensus_queue.cc:260] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: true } }
I20240503 08:11:41.341194 386 ts_tablet_manager.cc:1398] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240503 08:11:41.341866 397 tablet_bootstrap.cc:492] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:41.342244 30961 tablet_copy_service.cc:342] P ac3512789abc4bd890102fa6910f980b: Request end of tablet copy session 9f45625e8fff471382d1376ff8cf0d00-bee6e9222daa40e09dbf75c0ffaf479d received from {username='slave'} at 127.27.157.198:39269
I20240503 08:11:41.342346 397 tablet_bootstrap.cc:492] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00: Bootstrap complete.
I20240503 08:11:41.342417 30961 tablet_copy_service.cc:433] P ac3512789abc4bd890102fa6910f980b: ending tablet copy session 9f45625e8fff471382d1376ff8cf0d00-bee6e9222daa40e09dbf75c0ffaf479d on tablet bee6e9222daa40e09dbf75c0ffaf479d with peer 9f45625e8fff471382d1376ff8cf0d00
I20240503 08:11:41.342628 397 ts_tablet_manager.cc:1367] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00: Time spent bootstrapping tablet: real 0.215s user 0.060s sys 0.009s
I20240503 08:11:41.343938 397 raft_consensus.cc:348] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: true } }
I20240503 08:11:41.344192 397 raft_consensus.cc:729] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f45625e8fff471382d1376ff8cf0d00, State: Initialized, Role: LEARNER
I20240503 08:11:41.344473 397 consensus_queue.cc:260] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: true } }
I20240503 08:11:41.344928 397 ts_tablet_manager.cc:1398] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:11:41.345773 30961 tablet_copy_service.cc:342] P ac3512789abc4bd890102fa6910f980b: Request end of tablet copy session 9f45625e8fff471382d1376ff8cf0d00-fa28377977cf4ffa9347b5d45020c377 received from {username='slave'} at 127.27.157.198:39269
I20240503 08:11:41.345916 30961 tablet_copy_service.cc:433] P ac3512789abc4bd890102fa6910f980b: ending tablet copy session 9f45625e8fff471382d1376ff8cf0d00-fa28377977cf4ffa9347b5d45020c377 on tablet fa28377977cf4ffa9347b5d45020c377 with peer 9f45625e8fff471382d1376ff8cf0d00
I20240503 08:11:41.360491 32574 raft_consensus.cc:1203] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:41.401175 32041 raft_consensus.cc:1203] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:41.475914 32444 raft_consensus.cc:1203] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:41.494081 32172 raft_consensus.cc:1203] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:41.500252 32304 raft_consensus.cc:1203] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:41.543079 32444 raft_consensus.cc:1203] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:41.654947 32041 raft_consensus.cc:1203] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:41.672185 32710 raft_consensus.cc:1050] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: attempting to promote NON_VOTER a7b227b516a24a1b837415111eba1303 to VOTER
I20240503 08:11:41.673046 32710 consensus_queue.cc:237] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } }
I20240503 08:11:41.675144 32172 raft_consensus.cc:1261] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [term 1 LEARNER]: Refusing update from remote peer 3986faa56d004ea5b787a469ddb79c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:41.675863 468 consensus_queue.cc:1035] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:41.679716 32710 raft_consensus.cc:2932] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a7b227b516a24a1b837415111eba1303 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } } }
I20240503 08:11:41.680615 32172 raft_consensus.cc:2932] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a7b227b516a24a1b837415111eba1303 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } } }
I20240503 08:11:41.684145 30667 catalog_manager.cc:5484] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index 4 to 5, a7b227b516a24a1b837415111eba1303 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:41.760088 32574 raft_consensus.cc:1203] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:41.829939 30810 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35b0575c97db429fb6c30261552e090b"
dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46802
I20240503 08:11:41.830232 30810 raft_consensus.cc:595] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:41.874925 32710 raft_consensus.cc:1050] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: attempting to promote NON_VOTER a7b227b516a24a1b837415111eba1303 to VOTER
I20240503 08:11:41.875854 32710 consensus_queue.cc:237] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } }
I20240503 08:11:41.877959 32172 raft_consensus.cc:1261] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [term 1 LEARNER]: Refusing update from remote peer 3986faa56d004ea5b787a469ddb79c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:41.878576 468 consensus_queue.cc:1035] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:41.879743 457 raft_consensus.cc:1050] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: attempting to promote NON_VOTER 03e83fbce8ee4c5cacbf54f25e1f2665 to VOTER
I20240503 08:11:41.880633 457 consensus_queue.cc:237] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } }
I20240503 08:11:41.882704 32710 raft_consensus.cc:2932] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a7b227b516a24a1b837415111eba1303 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } } }
I20240503 08:11:41.883777 32172 raft_consensus.cc:2932] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a7b227b516a24a1b837415111eba1303 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } } }
I20240503 08:11:41.888175 30667 catalog_manager.cc:5484] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index 4 to 5, a7b227b516a24a1b837415111eba1303 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:41.893564 32574 raft_consensus.cc:1261] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 LEARNER]: Refusing update from remote peer ac3512789abc4bd890102fa6910f980b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:41.894573 457 consensus_queue.cc:1035] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:41.898495 32756 raft_consensus.cc:2932] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } } }
I20240503 08:11:41.899507 32574 raft_consensus.cc:2932] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } } }
I20240503 08:11:41.904201 30660 catalog_manager.cc:5484] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b reported cstate change: config changed from index 4 to 5, 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:41.952777 32756 raft_consensus.cc:1050] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: attempting to promote NON_VOTER 9f45625e8fff471382d1376ff8cf0d00 to VOTER
I20240503 08:11:41.953388 32756 consensus_queue.cc:237] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } }
I20240503 08:11:41.955698 32444 raft_consensus.cc:1261] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [term 1 LEARNER]: Refusing update from remote peer ac3512789abc4bd890102fa6910f980b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:41.956471 32756 consensus_queue.cc:1035] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:41.960803 32724 raft_consensus.cc:2932] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } } }
I20240503 08:11:41.961994 32444 raft_consensus.cc:2932] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } } }
I20240503 08:11:41.966099 30667 catalog_manager.cc:5484] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b reported cstate change: config changed from index 4 to 5, 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:41.992973 32724 raft_consensus.cc:1050] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: attempting to promote NON_VOTER a9b83be26ed14471b0335ffb28f12193 to VOTER
I20240503 08:11:41.993544 32724 consensus_queue.cc:237] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } }
I20240503 08:11:41.995638 32041 raft_consensus.cc:1261] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [term 1 LEARNER]: Refusing update from remote peer ac3512789abc4bd890102fa6910f980b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:41.996302 457 consensus_queue.cc:1035] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:42.000284 32724 raft_consensus.cc:2932] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9b83be26ed14471b0335ffb28f12193 (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } } }
I20240503 08:11:42.001243 32041 raft_consensus.cc:2932] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9b83be26ed14471b0335ffb28f12193 (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } } }
I20240503 08:11:42.005929 30660 catalog_manager.cc:5484] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b reported cstate change: config changed from index 4 to 5, a9b83be26ed14471b0335ffb28f12193 (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.057742 468 raft_consensus.cc:1050] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: attempting to promote NON_VOTER c5dc2c8f44d44889b3888f6265d07de9 to VOTER
I20240503 08:11:42.058387 468 consensus_queue.cc:237] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false } }
I20240503 08:11:42.060611 32304 raft_consensus.cc:1261] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [term 1 LEARNER]: Refusing update from remote peer 3986faa56d004ea5b787a469ddb79c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:42.061354 32710 consensus_queue.cc:1035] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:42.064488 468 raft_consensus.cc:2932] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5dc2c8f44d44889b3888f6265d07de9 (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false } } }
I20240503 08:11:42.065354 32304 raft_consensus.cc:2932] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5dc2c8f44d44889b3888f6265d07de9 (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false } } }
I20240503 08:11:42.070225 30667 catalog_manager.cc:5484] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index 4 to 5, c5dc2c8f44d44889b3888f6265d07de9 (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.078825 451 raft_consensus.cc:979] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: : Instructing follower a7b227b516a24a1b837415111eba1303 to start an election
I20240503 08:11:42.079036 451 raft_consensus.cc:1067] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Signalling peer a7b227b516a24a1b837415111eba1303 to start an election
I20240503 08:11:42.080178 32172 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35b0575c97db429fb6c30261552e090b"
dest_uuid: "a7b227b516a24a1b837415111eba1303"
from {username='slave'} at 127.27.157.193:41279
I20240503 08:11:42.080560 32172 raft_consensus.cc:482] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:42.080780 32172 raft_consensus.cc:3037] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.083918 32172 raft_consensus.cc:504] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } }
I20240503 08:11:42.085641 32172 leader_election.cc:288] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [CANDIDATE]: Term 2 election: Requested vote from peers 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:42.086530 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b0575c97db429fb6c30261552e090b" candidate_uuid: "a7b227b516a24a1b837415111eba1303" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
I20240503 08:11:42.086894 30810 raft_consensus.cc:3032] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:42.087002 30810 raft_consensus.cc:729] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:42.087699 30810 consensus_queue.cc:260] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } }
I20240503 08:11:42.088128 30810 raft_consensus.cc:3037] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.091063 30810 raft_consensus.cc:2454] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7b227b516a24a1b837415111eba1303 in term 2.
I20240503 08:11:42.091903 32106 leader_election.cc:302] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81, a7b227b516a24a1b837415111eba1303; no voters:
I20240503 08:11:42.092648 353 raft_consensus.cc:2781] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:42.098184 353 raft_consensus.cc:686] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [term 2 LEADER]: Becoming Leader. State: Replica: a7b227b516a24a1b837415111eba1303, State: Running, Role: LEADER
I20240503 08:11:42.099061 353 consensus_queue.cc:237] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } }
I20240503 08:11:42.105296 30660 catalog_manager.cc:5484] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 reported cstate change: term changed from 1 to 2, leader changed from 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) to a7b227b516a24a1b837415111eba1303 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "a7b227b516a24a1b837415111eba1303" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.105670 32724 raft_consensus.cc:1050] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: attempting to promote NON_VOTER 9f45625e8fff471382d1376ff8cf0d00 to VOTER
I20240503 08:11:42.106360 32724 consensus_queue.cc:237] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } }
I20240503 08:11:42.108392 32444 raft_consensus.cc:1261] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [term 1 LEARNER]: Refusing update from remote peer ac3512789abc4bd890102fa6910f980b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:42.108942 457 consensus_queue.cc:1035] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:42.113816 32724 raft_consensus.cc:2932] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } } }
I20240503 08:11:42.119091 32444 raft_consensus.cc:2932] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } } }
I20240503 08:11:42.120355 30941 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e188dd3547440b183799c9a796b4a85"
dest_uuid: "ac3512789abc4bd890102fa6910f980b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55200
I20240503 08:11:42.120640 30941 raft_consensus.cc:595] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.122328 30660 catalog_manager.cc:5484] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b reported cstate change: config changed from index 4 to 5, 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:11:42.133801 30654 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 35b0575c97db429fb6c30261552e090b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader a7b227b516a24a1b837415111eba1303 (127.27.157.196:42725) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 08:11:42.145924 451 raft_consensus.cc:1050] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: attempting to promote NON_VOTER 03e83fbce8ee4c5cacbf54f25e1f2665 to VOTER
I20240503 08:11:42.146617 451 consensus_queue.cc:237] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } }
I20240503 08:11:42.146905 30941 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bee6e9222daa40e09dbf75c0ffaf479d"
dest_uuid: "ac3512789abc4bd890102fa6910f980b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55210
I20240503 08:11:42.147089 30941 raft_consensus.cc:595] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.150996 32574 raft_consensus.cc:1261] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 LEARNER]: Refusing update from remote peer 3986faa56d004ea5b787a469ddb79c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:42.151768 451 consensus_queue.cc:1035] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:42.155027 468 raft_consensus.cc:2932] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } } }
I20240503 08:11:42.155721 32574 raft_consensus.cc:2932] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } } }
I20240503 08:11:42.159838 451 raft_consensus.cc:1050] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: attempting to promote NON_VOTER a9b83be26ed14471b0335ffb28f12193 to VOTER
I20240503 08:11:42.160251 30660 catalog_manager.cc:5484] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index 4 to 5, 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.160601 451 consensus_queue.cc:237] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } }
I20240503 08:11:42.163218 32041 raft_consensus.cc:1261] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [term 1 LEARNER]: Refusing update from remote peer 3986faa56d004ea5b787a469ddb79c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:42.164060 451 consensus_queue.cc:1035] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:42.167593 468 raft_consensus.cc:2932] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9b83be26ed14471b0335ffb28f12193 (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } } }
I20240503 08:11:42.171221 32041 raft_consensus.cc:2932] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a9b83be26ed14471b0335ffb28f12193 (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } } }
I20240503 08:11:42.172346 30667 catalog_manager.cc:5484] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index 4 to 5, a9b83be26ed14471b0335ffb28f12193 (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.177158 30941 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa28377977cf4ffa9347b5d45020c377"
dest_uuid: "ac3512789abc4bd890102fa6910f980b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55220
I20240503 08:11:42.177392 30941 raft_consensus.cc:595] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.209007 30810 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "153c5b78de3342eab81fdce1a7f199d5"
dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46858
I20240503 08:11:42.209246 30810 raft_consensus.cc:595] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.228328 30810 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b123f5c42af4d11b4325b3cd6226c29"
dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46878
I20240503 08:11:42.228682 30810 raft_consensus.cc:595] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.247918 30810 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b5f4bb2e73c4643a964b3c834b820aa"
dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46894
I20240503 08:11:42.248172 30810 raft_consensus.cc:595] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.267582 30941 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20b5e4e7b78f4f0d979b936d92dd1eb3"
dest_uuid: "ac3512789abc4bd890102fa6910f980b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55244
I20240503 08:11:42.267807 30941 raft_consensus.cc:595] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.286885 30810 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1f4d7f23456f4bddabcdd74ddea40a33"
dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46920
I20240503 08:11:42.287144 30810 raft_consensus.cc:595] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.453749 468 raft_consensus.cc:979] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: : Instructing follower a7b227b516a24a1b837415111eba1303 to start an election
I20240503 08:11:42.454010 451 raft_consensus.cc:1067] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Signalling peer a7b227b516a24a1b837415111eba1303 to start an election
I20240503 08:11:42.454810 32171 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "153c5b78de3342eab81fdce1a7f199d5"
dest_uuid: "a7b227b516a24a1b837415111eba1303"
from {username='slave'} at 127.27.157.193:41279
I20240503 08:11:42.455063 32171 raft_consensus.cc:482] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:42.455215 32171 raft_consensus.cc:3037] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.458293 32171 raft_consensus.cc:504] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } }
I20240503 08:11:42.458998 32171 leader_election.cc:288] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [CANDIDATE]: Term 2 election: Requested vote from peers 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:42.459776 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153c5b78de3342eab81fdce1a7f199d5" candidate_uuid: "a7b227b516a24a1b837415111eba1303" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
I20240503 08:11:42.460007 30810 raft_consensus.cc:3032] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:42.460088 30810 raft_consensus.cc:729] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:42.460420 30810 consensus_queue.cc:260] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } }
I20240503 08:11:42.460781 30810 raft_consensus.cc:3037] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.462941 30810 raft_consensus.cc:2454] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7b227b516a24a1b837415111eba1303 in term 2.
I20240503 08:11:42.463518 32106 leader_election.cc:302] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81, a7b227b516a24a1b837415111eba1303; no voters:
I20240503 08:11:42.463886 353 raft_consensus.cc:2781] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:42.464064 353 raft_consensus.cc:686] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [term 2 LEADER]: Becoming Leader. State: Replica: a7b227b516a24a1b837415111eba1303, State: Running, Role: LEADER
I20240503 08:11:42.464306 353 consensus_queue.cc:237] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } }
I20240503 08:11:42.466745 30660 catalog_manager.cc:5484] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 reported cstate change: term changed from 1 to 2, leader changed from 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) to a7b227b516a24a1b837415111eba1303 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "a7b227b516a24a1b837415111eba1303" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.471668 457 raft_consensus.cc:979] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: : Instructing follower 03e83fbce8ee4c5cacbf54f25e1f2665 to start an election
I20240503 08:11:42.471868 457 raft_consensus.cc:1067] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Signalling peer 03e83fbce8ee4c5cacbf54f25e1f2665 to start an election
I20240503 08:11:42.472730 32574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20b5e4e7b78f4f0d979b936d92dd1eb3"
dest_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665"
from {username='slave'} at 127.27.157.194:41289
I20240503 08:11:42.472987 32574 raft_consensus.cc:482] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:42.473150 32574 raft_consensus.cc:3037] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.475116 32574 raft_consensus.cc:504] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } }
I20240503 08:11:42.476590 32574 leader_election.cc:288] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [CANDIDATE]: Term 2 election: Requested vote from peers ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429)
I20240503 08:11:42.477351 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b5e4e7b78f4f0d979b936d92dd1eb3" candidate_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ac3512789abc4bd890102fa6910f980b"
I20240503 08:11:42.477656 30941 raft_consensus.cc:3032] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:42.477777 30941 raft_consensus.cc:729] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:42.478086 30941 consensus_queue.cc:260] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } }
I20240503 08:11:42.478427 30941 raft_consensus.cc:3037] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.480651 30941 raft_consensus.cc:2454] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e83fbce8ee4c5cacbf54f25e1f2665 in term 2.
I20240503 08:11:42.481408 32510 leader_election.cc:302] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 03e83fbce8ee4c5cacbf54f25e1f2665, ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:42.481832 454 raft_consensus.cc:2781] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:42.482306 454 raft_consensus.cc:686] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 2 LEADER]: Becoming Leader. State: Replica: 03e83fbce8ee4c5cacbf54f25e1f2665, State: Running, Role: LEADER
I20240503 08:11:42.482885 454 consensus_queue.cc:237] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } }
I20240503 08:11:42.486697 30660 catalog_manager.cc:5484] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 reported cstate change: term changed from 1 to 2, leader changed from ac3512789abc4bd890102fa6910f980b (127.27.157.194) to 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199). New cstate: current_term: 2 leader_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.497135 451 raft_consensus.cc:979] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: : Instructing follower c5dc2c8f44d44889b3888f6265d07de9 to start an election
I20240503 08:11:42.497373 451 raft_consensus.cc:1067] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Signalling peer c5dc2c8f44d44889b3888f6265d07de9 to start an election
I20240503 08:11:42.498495 32304 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1f4d7f23456f4bddabcdd74ddea40a33"
dest_uuid: "c5dc2c8f44d44889b3888f6265d07de9"
from {username='slave'} at 127.27.157.193:46921
I20240503 08:11:42.498898 32304 raft_consensus.cc:482] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:42.499054 32304 raft_consensus.cc:3037] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.503286 32304 raft_consensus.cc:504] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false } }
I20240503 08:11:42.505131 32304 leader_election.cc:288] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [CANDIDATE]: Term 2 election: Requested vote from peers 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:42.506121 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4d7f23456f4bddabcdd74ddea40a33" candidate_uuid: "c5dc2c8f44d44889b3888f6265d07de9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
I20240503 08:11:42.506414 30810 raft_consensus.cc:3032] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:42.506506 30810 raft_consensus.cc:729] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:42.506819 30810 consensus_queue.cc:260] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false } }
I20240503 08:11:42.507114 30810 raft_consensus.cc:3037] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.509181 30810 raft_consensus.cc:2454] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5dc2c8f44d44889b3888f6265d07de9 in term 2.
I20240503 08:11:42.510018 32236 leader_election.cc:302] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81, c5dc2c8f44d44889b3888f6265d07de9; no voters:
I20240503 08:11:42.510419 570 raft_consensus.cc:2781] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:42.510907 570 raft_consensus.cc:686] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [term 2 LEADER]: Becoming Leader. State: Replica: c5dc2c8f44d44889b3888f6265d07de9, State: Running, Role: LEADER
I20240503 08:11:42.511262 570 consensus_queue.cc:237] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false } }
I20240503 08:11:42.514340 30660 catalog_manager.cc:5484] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 reported cstate change: term changed from 1 to 2, leader changed from 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) to c5dc2c8f44d44889b3888f6265d07de9 (127.27.157.197). New cstate: current_term: 2 leader_uuid: "c5dc2c8f44d44889b3888f6265d07de9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.526651 30941 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e188dd3547440b183799c9a796b4a85"
dest_uuid: "ac3512789abc4bd890102fa6910f980b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55264
I20240503 08:11:42.526926 30941 raft_consensus.cc:595] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.529968 457 raft_consensus.cc:979] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: : Instructing follower 9f45625e8fff471382d1376ff8cf0d00 to start an election
I20240503 08:11:42.530175 32756 raft_consensus.cc:1067] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Signalling peer 9f45625e8fff471382d1376ff8cf0d00 to start an election
I20240503 08:11:42.531011 32444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa28377977cf4ffa9347b5d45020c377"
dest_uuid: "9f45625e8fff471382d1376ff8cf0d00"
from {username='slave'} at 127.27.157.194:39179
I20240503 08:11:42.531358 32444 raft_consensus.cc:482] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:42.531581 32444 raft_consensus.cc:3037] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.534624 32444 raft_consensus.cc:504] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } }
I20240503 08:11:42.536283 32444 leader_election.cc:288] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [CANDIDATE]: Term 2 election: Requested vote from peers ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429)
I20240503 08:11:42.536926 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa28377977cf4ffa9347b5d45020c377" candidate_uuid: "9f45625e8fff471382d1376ff8cf0d00" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ac3512789abc4bd890102fa6910f980b"
I20240503 08:11:42.537209 30941 raft_consensus.cc:3032] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:42.537330 30941 raft_consensus.cc:729] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:42.537635 30941 consensus_queue.cc:260] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } }
I20240503 08:11:42.538012 30941 raft_consensus.cc:3037] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.540510 30941 raft_consensus.cc:2454] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f45625e8fff471382d1376ff8cf0d00 in term 2.
I20240503 08:11:42.541214 32380 leader_election.cc:302] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9f45625e8fff471382d1376ff8cf0d00, ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:42.541695 456 raft_consensus.cc:2781] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:42.543646 456 raft_consensus.cc:686] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [term 2 LEADER]: Becoming Leader. State: Replica: 9f45625e8fff471382d1376ff8cf0d00, State: Running, Role: LEADER
I20240503 08:11:42.543713 30810 raft_consensus.cc:1261] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Refusing update from remote peer a7b227b516a24a1b837415111eba1303: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:42.544359 456 consensus_queue.cc:237] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } }
I20240503 08:11:42.544647 576 consensus_queue.cc:1035] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:42.548712 30667 catalog_manager.cc:5484] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 reported cstate change: term changed from 1 to 2, leader changed from ac3512789abc4bd890102fa6910f980b (127.27.157.194) to 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198). New cstate: current_term: 2 leader_uuid: "9f45625e8fff471382d1376ff8cf0d00" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.555291 30941 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bee6e9222daa40e09dbf75c0ffaf479d"
dest_uuid: "ac3512789abc4bd890102fa6910f980b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55288
I20240503 08:11:42.555580 30941 raft_consensus.cc:595] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.587904 32724 raft_consensus.cc:979] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: : Instructing follower a9b83be26ed14471b0335ffb28f12193 to start an election
I20240503 08:11:42.588121 32724 raft_consensus.cc:1067] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Signalling peer a9b83be26ed14471b0335ffb28f12193 to start an election
I20240503 08:11:42.589603 32041 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e188dd3547440b183799c9a796b4a85"
dest_uuid: "a9b83be26ed14471b0335ffb28f12193"
from {username='slave'} at 127.27.157.194:55547
I20240503 08:11:42.589951 32041 raft_consensus.cc:482] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:42.590168 32041 raft_consensus.cc:3037] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.593058 32041 raft_consensus.cc:504] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } }
I20240503 08:11:42.594677 32041 leader_election.cc:288] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [CANDIDATE]: Term 2 election: Requested vote from peers ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429)
I20240503 08:11:42.595480 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e188dd3547440b183799c9a796b4a85" candidate_uuid: "a9b83be26ed14471b0335ffb28f12193" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ac3512789abc4bd890102fa6910f980b"
I20240503 08:11:42.595703 30941 raft_consensus.cc:3032] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:42.595808 30941 raft_consensus.cc:729] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:42.596133 30941 consensus_queue.cc:260] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } }
I20240503 08:11:42.596506 30941 raft_consensus.cc:3037] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.599002 30941 raft_consensus.cc:2454] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9b83be26ed14471b0335ffb28f12193 in term 2.
I20240503 08:11:42.599855 31977 leader_election.cc:302] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a9b83be26ed14471b0335ffb28f12193, ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:42.600368 449 raft_consensus.cc:2781] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:42.600919 449 raft_consensus.cc:686] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [term 2 LEADER]: Becoming Leader. State: Replica: a9b83be26ed14471b0335ffb28f12193, State: Running, Role: LEADER
I20240503 08:11:42.601509 449 consensus_queue.cc:237] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } }
I20240503 08:11:42.606565 30667 catalog_manager.cc:5484] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 reported cstate change: term changed from 1 to 2, leader changed from ac3512789abc4bd890102fa6910f980b (127.27.157.194) to a9b83be26ed14471b0335ffb28f12193 (127.27.157.195). New cstate: current_term: 2 leader_uuid: "a9b83be26ed14471b0335ffb28f12193" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.620123 30810 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b123f5c42af4d11b4325b3cd6226c29"
dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46952
I20240503 08:11:42.620416 30810 raft_consensus.cc:595] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.640743 32756 raft_consensus.cc:979] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: : Instructing follower 9f45625e8fff471382d1376ff8cf0d00 to start an election
I20240503 08:11:42.640929 32756 raft_consensus.cc:1067] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Signalling peer 9f45625e8fff471382d1376ff8cf0d00 to start an election
I20240503 08:11:42.641670 32444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bee6e9222daa40e09dbf75c0ffaf479d"
dest_uuid: "9f45625e8fff471382d1376ff8cf0d00"
from {username='slave'} at 127.27.157.194:39179
I20240503 08:11:42.641924 32444 raft_consensus.cc:482] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:42.642047 32444 raft_consensus.cc:3037] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.644006 30810 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b5f4bb2e73c4643a964b3c834b820aa"
dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38256
I20240503 08:11:42.644186 30810 raft_consensus.cc:595] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:42.644589 32444 raft_consensus.cc:504] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } }
I20240503 08:11:42.646427 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee6e9222daa40e09dbf75c0ffaf479d" candidate_uuid: "9f45625e8fff471382d1376ff8cf0d00" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ac3512789abc4bd890102fa6910f980b"
I20240503 08:11:42.646713 30941 raft_consensus.cc:3032] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:42.646835 30941 raft_consensus.cc:729] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:42.647151 30941 consensus_queue.cc:260] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } }
I20240503 08:11:42.647493 30941 raft_consensus.cc:3037] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.647528 32444 leader_election.cc:288] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [CANDIDATE]: Term 2 election: Requested vote from peers ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429)
I20240503 08:11:42.651103 30941 raft_consensus.cc:2454] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f45625e8fff471382d1376ff8cf0d00 in term 2.
I20240503 08:11:42.651863 32380 leader_election.cc:302] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9f45625e8fff471382d1376ff8cf0d00, ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:42.652287 456 raft_consensus.cc:2781] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:42.652494 456 raft_consensus.cc:686] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [term 2 LEADER]: Becoming Leader. State: Replica: 9f45625e8fff471382d1376ff8cf0d00, State: Running, Role: LEADER
I20240503 08:11:42.652781 456 consensus_queue.cc:237] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } }
I20240503 08:11:42.656953 30667 catalog_manager.cc:5484] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 reported cstate change: term changed from 1 to 2, leader changed from ac3512789abc4bd890102fa6910f980b (127.27.157.194) to 9f45625e8fff471382d1376ff8cf0d00 (127.27.157.198). New cstate: current_term: 2 leader_uuid: "9f45625e8fff471382d1376ff8cf0d00" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.716279 451 raft_consensus.cc:979] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: : Instructing follower 03e83fbce8ee4c5cacbf54f25e1f2665 to start an election
I20240503 08:11:42.716491 451 raft_consensus.cc:1067] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Signalling peer 03e83fbce8ee4c5cacbf54f25e1f2665 to start an election
I20240503 08:11:42.717363 32574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b5f4bb2e73c4643a964b3c834b820aa"
dest_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665"
from {username='slave'} at 127.27.157.193:52531
I20240503 08:11:42.717618 32574 raft_consensus.cc:482] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:42.717728 32574 raft_consensus.cc:3037] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.720077 32574 raft_consensus.cc:504] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } }
I20240503 08:11:42.720882 32574 leader_election.cc:288] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [CANDIDATE]: Term 2 election: Requested vote from peers 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:42.721381 451 raft_consensus.cc:979] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: : Instructing follower a9b83be26ed14471b0335ffb28f12193 to start an election
I20240503 08:11:42.721556 451 raft_consensus.cc:1067] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Signalling peer a9b83be26ed14471b0335ffb28f12193 to start an election
I20240503 08:11:42.721531 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5f4bb2e73c4643a964b3c834b820aa" candidate_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
I20240503 08:11:42.721781 30810 raft_consensus.cc:3032] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:42.721908 30810 raft_consensus.cc:729] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:42.722201 30810 consensus_queue.cc:260] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } }
I20240503 08:11:42.722507 30810 raft_consensus.cc:3037] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.723809 32041 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b123f5c42af4d11b4325b3cd6226c29"
dest_uuid: "a9b83be26ed14471b0335ffb28f12193"
from {username='slave'} at 127.27.157.193:37377
I20240503 08:11:42.724110 32041 raft_consensus.cc:482] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:42.724254 32041 raft_consensus.cc:3037] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.725055 30810 raft_consensus.cc:2454] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e83fbce8ee4c5cacbf54f25e1f2665 in term 2.
I20240503 08:11:42.725636 32508 leader_election.cc:302] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 03e83fbce8ee4c5cacbf54f25e1f2665, 3986faa56d004ea5b787a469ddb79c81; no voters:
I20240503 08:11:42.726050 454 raft_consensus.cc:2781] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:42.726212 454 raft_consensus.cc:686] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 2 LEADER]: Becoming Leader. State: Replica: 03e83fbce8ee4c5cacbf54f25e1f2665, State: Running, Role: LEADER
I20240503 08:11:42.726446 454 consensus_queue.cc:237] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } }
I20240503 08:11:42.727047 32041 raft_consensus.cc:504] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } }
I20240503 08:11:42.727836 32041 leader_election.cc:288] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [CANDIDATE]: Term 2 election: Requested vote from peers 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:42.728543 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b123f5c42af4d11b4325b3cd6226c29" candidate_uuid: "a9b83be26ed14471b0335ffb28f12193" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
I20240503 08:11:42.728746 30810 raft_consensus.cc:3032] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:42.728876 30810 raft_consensus.cc:729] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:42.729161 30810 consensus_queue.cc:260] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } }
I20240503 08:11:42.729480 30810 raft_consensus.cc:3037] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:42.729383 30667 catalog_manager.cc:5484] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 reported cstate change: term changed from 1 to 2, leader changed from 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) to 03e83fbce8ee4c5cacbf54f25e1f2665 (127.27.157.199). New cstate: current_term: 2 leader_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.731618 30810 raft_consensus.cc:2454] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9b83be26ed14471b0335ffb28f12193 in term 2.
I20240503 08:11:42.732151 31975 leader_election.cc:302] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81, a9b83be26ed14471b0335ffb28f12193; no voters:
I20240503 08:11:42.732461 449 raft_consensus.cc:2781] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:42.732646 449 raft_consensus.cc:686] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [term 2 LEADER]: Becoming Leader. State: Replica: a9b83be26ed14471b0335ffb28f12193, State: Running, Role: LEADER
I20240503 08:11:42.732925 449 consensus_queue.cc:237] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } }
I20240503 08:11:42.735256 30667 catalog_manager.cc:5484] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 reported cstate change: term changed from 1 to 2, leader changed from 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) to a9b83be26ed14471b0335ffb28f12193 (127.27.157.195). New cstate: current_term: 2 leader_uuid: "a9b83be26ed14471b0335ffb28f12193" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.807642 32171 consensus_queue.cc:237] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } }
I20240503 08:11:42.808610 353 raft_consensus.cc:2932] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } } }
I20240503 08:11:42.811884 30654 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 35b0575c97db429fb6c30261552e090b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:11:42.813385 30667 catalog_manager.cc:5484] T 35b0575c97db429fb6c30261552e090b P a7b227b516a24a1b837415111eba1303 reported cstate change: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New cstate: current_term: 2 leader_uuid: "a7b227b516a24a1b837415111eba1303" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:42.817651 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b0575c97db429fb6c30261552e090b with delete_type TABLET_DATA_TOMBSTONED (TS 3986faa56d004ea5b787a469ddb79c81 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53736
I20240503 08:11:42.818574 741 tablet_replica.cc:330] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: stopping tablet replica
I20240503 08:11:42.819023 741 raft_consensus.cc:2229] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:42.819375 741 raft_consensus.cc:2258] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:42.820863 741 ts_tablet_manager.cc:1875] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:42.826597 741 ts_tablet_manager.cc:1888] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:42.826793 741 log.cc:1196] T 35b0575c97db429fb6c30261552e090b P 3986faa56d004ea5b787a469ddb79c81: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/35b0575c97db429fb6c30261552e090b
I20240503 08:11:42.827862 30654 catalog_manager.cc:4830] TS 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943): tablet 35b0575c97db429fb6c30261552e090b (table rebalance_test_table_4 [id=9a0826219eb944058c62c4f29729bf0e]) successfully deleted
I20240503 08:11:42.903036 30810 raft_consensus.cc:1261] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Refusing update from remote peer a7b227b516a24a1b837415111eba1303: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:42.904024 353 consensus_queue.cc:1035] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:42.905124 30941 raft_consensus.cc:1261] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Refusing update from remote peer 03e83fbce8ee4c5cacbf54f25e1f2665: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:42.906615 666 consensus_queue.cc:1035] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:42.996434 30941 raft_consensus.cc:1261] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Refusing update from remote peer a9b83be26ed14471b0335ffb28f12193: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:42.997208 449 consensus_queue.cc:1035] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:43.041606 30941 raft_consensus.cc:1261] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Refusing update from remote peer 9f45625e8fff471382d1376ff8cf0d00: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:43.042500 689 consensus_queue.cc:1035] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:43.083391 32574 consensus_queue.cc:237] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } }
I20240503 08:11:43.084461 666 raft_consensus.cc:2932] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ac3512789abc4bd890102fa6910f980b (127.27.157.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } } }
I20240503 08:11:43.088497 30653 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 20b5e4e7b78f4f0d979b936d92dd1eb3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:11:43.089766 30667 catalog_manager.cc:5484] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P 03e83fbce8ee4c5cacbf54f25e1f2665 reported cstate change: config changed from index 5 to 7, VOTER ac3512789abc4bd890102fa6910f980b (127.27.157.194) evicted. New cstate: current_term: 2 leader_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:43.093863 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b5e4e7b78f4f0d979b936d92dd1eb3 with delete_type TABLET_DATA_TOMBSTONED (TS ac3512789abc4bd890102fa6910f980b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59376
I20240503 08:11:43.094717 833 tablet_replica.cc:330] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: stopping tablet replica
I20240503 08:11:43.095199 833 raft_consensus.cc:2229] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:43.095489 833 raft_consensus.cc:2258] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:43.096982 833 ts_tablet_manager.cc:1875] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:43.102543 833 ts_tablet_manager.cc:1888] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:43.102712 833 log.cc:1196] T 20b5e4e7b78f4f0d979b936d92dd1eb3 P ac3512789abc4bd890102fa6910f980b: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/20b5e4e7b78f4f0d979b936d92dd1eb3
I20240503 08:11:43.103730 30656 catalog_manager.cc:4830] TS ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429): tablet 20b5e4e7b78f4f0d979b936d92dd1eb3 (table rebalance_test_table_1 [id=05f1bef55b064e9783e456fbd220acb4]) successfully deleted
I20240503 08:11:43.124804 30810 raft_consensus.cc:1261] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Refusing update from remote peer 03e83fbce8ee4c5cacbf54f25e1f2665: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:43.125496 666 consensus_queue.cc:1035] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:43.130753 30810 raft_consensus.cc:1261] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Refusing update from remote peer c5dc2c8f44d44889b3888f6265d07de9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:43.131582 835 consensus_queue.cc:1035] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:43.134354 32171 consensus_queue.cc:237] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } }
I20240503 08:11:43.135262 576 raft_consensus.cc:2932] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } } }
I20240503 08:11:43.140005 30654 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 153c5b78de3342eab81fdce1a7f199d5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:11:43.141207 30669 catalog_manager.cc:5484] T 153c5b78de3342eab81fdce1a7f199d5 P a7b227b516a24a1b837415111eba1303 reported cstate change: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New cstate: current_term: 2 leader_uuid: "a7b227b516a24a1b837415111eba1303" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7b227b516a24a1b837415111eba1303" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42725 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:43.145022 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 153c5b78de3342eab81fdce1a7f199d5 with delete_type TABLET_DATA_TOMBSTONED (TS 3986faa56d004ea5b787a469ddb79c81 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53736
I20240503 08:11:43.145386 741 tablet_replica.cc:330] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: stopping tablet replica
I20240503 08:11:43.145571 741 raft_consensus.cc:2229] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:43.145785 741 raft_consensus.cc:2258] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:43.146764 741 ts_tablet_manager.cc:1875] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:43.151367 741 ts_tablet_manager.cc:1888] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:43.151518 741 log.cc:1196] T 153c5b78de3342eab81fdce1a7f199d5 P 3986faa56d004ea5b787a469ddb79c81: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/153c5b78de3342eab81fdce1a7f199d5
I20240503 08:11:43.152308 30654 catalog_manager.cc:4830] TS 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943): tablet 153c5b78de3342eab81fdce1a7f199d5 (table rebalance_test_table_2 [id=48d5c85892af46148fda788532045ac1]) successfully deleted
I20240503 08:11:43.163923 32304 consensus_queue.cc:237] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false } }
I20240503 08:11:43.164929 839 raft_consensus.cc:2932] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false } } }
I20240503 08:11:43.167666 30941 raft_consensus.cc:1261] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Refusing update from remote peer 9f45625e8fff471382d1376ff8cf0d00: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:43.168387 30653 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f4d7f23456f4bddabcdd74ddea40a33 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:11:43.168602 689 consensus_queue.cc:1035] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:43.170177 30669 catalog_manager.cc:5484] T 1f4d7f23456f4bddabcdd74ddea40a33 P c5dc2c8f44d44889b3888f6265d07de9 reported cstate change: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New cstate: current_term: 2 leader_uuid: "c5dc2c8f44d44889b3888f6265d07de9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c5dc2c8f44d44889b3888f6265d07de9" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 34939 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:43.175297 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4d7f23456f4bddabcdd74ddea40a33 with delete_type TABLET_DATA_TOMBSTONED (TS 3986faa56d004ea5b787a469ddb79c81 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53736
I20240503 08:11:43.175823 741 tablet_replica.cc:330] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: stopping tablet replica
I20240503 08:11:43.176061 741 raft_consensus.cc:2229] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:43.176303 741 raft_consensus.cc:2258] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:43.177135 741 ts_tablet_manager.cc:1875] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:43.181694 741 ts_tablet_manager.cc:1888] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:43.181833 741 log.cc:1196] T 1f4d7f23456f4bddabcdd74ddea40a33 P 3986faa56d004ea5b787a469ddb79c81: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/1f4d7f23456f4bddabcdd74ddea40a33
I20240503 08:11:43.182528 30654 catalog_manager.cc:4830] TS 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943): tablet 1f4d7f23456f4bddabcdd74ddea40a33 (table rebalance_test_table_0 [id=b355892f4af64661884a4ba55276ce1c]) successfully deleted
I20240503 08:11:43.212097 32444 consensus_queue.cc:237] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } }
I20240503 08:11:43.213210 841 raft_consensus.cc:2932] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ac3512789abc4bd890102fa6910f980b (127.27.157.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } } }
I20240503 08:11:43.216498 30655 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fa28377977cf4ffa9347b5d45020c377 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:11:43.217912 30660 catalog_manager.cc:5484] T fa28377977cf4ffa9347b5d45020c377 P 9f45625e8fff471382d1376ff8cf0d00 reported cstate change: config changed from index 5 to 7, VOTER ac3512789abc4bd890102fa6910f980b (127.27.157.194) evicted. New cstate: current_term: 2 leader_uuid: "9f45625e8fff471382d1376ff8cf0d00" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:43.221483 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet fa28377977cf4ffa9347b5d45020c377 with delete_type TABLET_DATA_TOMBSTONED (TS ac3512789abc4bd890102fa6910f980b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59376
I20240503 08:11:43.221870 833 tablet_replica.cc:330] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: stopping tablet replica
I20240503 08:11:43.222076 833 raft_consensus.cc:2229] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:43.222271 833 raft_consensus.cc:2258] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:43.223094 833 ts_tablet_manager.cc:1875] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:43.227784 833 ts_tablet_manager.cc:1888] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:43.227938 833 log.cc:1196] T fa28377977cf4ffa9347b5d45020c377 P ac3512789abc4bd890102fa6910f980b: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/fa28377977cf4ffa9347b5d45020c377
I20240503 08:11:43.228789 30656 catalog_manager.cc:4830] TS ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429): tablet fa28377977cf4ffa9347b5d45020c377 (table rebalance_test_table_3 [id=4074a9f3638e495cb809de913789b611]) successfully deleted
I20240503 08:11:43.246608 30810 raft_consensus.cc:1261] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Refusing update from remote peer a9b83be26ed14471b0335ffb28f12193: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:43.247308 449 consensus_queue.cc:1035] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:43.255852 32041 consensus_queue.cc:237] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } }
I20240503 08:11:43.256810 449 raft_consensus.cc:2932] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ac3512789abc4bd890102fa6910f980b (127.27.157.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } } }
I20240503 08:11:43.259718 30656 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e188dd3547440b183799c9a796b4a85 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:11:43.261085 30667 catalog_manager.cc:5484] T 6e188dd3547440b183799c9a796b4a85 P a9b83be26ed14471b0335ffb28f12193 reported cstate change: config changed from index 5 to 7, VOTER ac3512789abc4bd890102fa6910f980b (127.27.157.194) evicted. New cstate: current_term: 2 leader_uuid: "a9b83be26ed14471b0335ffb28f12193" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:43.264655 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e188dd3547440b183799c9a796b4a85 with delete_type TABLET_DATA_TOMBSTONED (TS ac3512789abc4bd890102fa6910f980b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59376
I20240503 08:11:43.265043 833 tablet_replica.cc:330] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: stopping tablet replica
I20240503 08:11:43.265213 833 raft_consensus.cc:2229] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:43.265378 833 raft_consensus.cc:2258] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:43.266134 833 ts_tablet_manager.cc:1875] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:43.271186 833 ts_tablet_manager.cc:1888] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:43.271353 833 log.cc:1196] T 6e188dd3547440b183799c9a796b4a85 P ac3512789abc4bd890102fa6910f980b: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/6e188dd3547440b183799c9a796b4a85
I20240503 08:11:43.272143 30656 catalog_manager.cc:4830] TS ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429): tablet 6e188dd3547440b183799c9a796b4a85 (table rebalance_test_table_1 [id=05f1bef55b064e9783e456fbd220acb4]) successfully deleted
I20240503 08:11:43.280629 32444 consensus_queue.cc:237] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } }
I20240503 08:11:43.281569 841 raft_consensus.cc:2932] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ac3512789abc4bd890102fa6910f980b (127.27.157.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } } }
I20240503 08:11:43.284379 30655 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bee6e9222daa40e09dbf75c0ffaf479d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:11:43.285768 30667 catalog_manager.cc:5484] T bee6e9222daa40e09dbf75c0ffaf479d P 9f45625e8fff471382d1376ff8cf0d00 reported cstate change: config changed from index 5 to 7, VOTER ac3512789abc4bd890102fa6910f980b (127.27.157.194) evicted. New cstate: current_term: 2 leader_uuid: "9f45625e8fff471382d1376ff8cf0d00" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9f45625e8fff471382d1376ff8cf0d00" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 42339 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:43.289290 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet bee6e9222daa40e09dbf75c0ffaf479d with delete_type TABLET_DATA_TOMBSTONED (TS ac3512789abc4bd890102fa6910f980b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59376
I20240503 08:11:43.289650 833 tablet_replica.cc:330] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: stopping tablet replica
I20240503 08:11:43.289819 833 raft_consensus.cc:2229] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:43.289979 833 raft_consensus.cc:2258] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:43.290822 833 ts_tablet_manager.cc:1875] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:43.295925 833 ts_tablet_manager.cc:1888] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:43.296082 833 log.cc:1196] T bee6e9222daa40e09dbf75c0ffaf479d P ac3512789abc4bd890102fa6910f980b: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/bee6e9222daa40e09dbf75c0ffaf479d
I20240503 08:11:43.296994 30656 catalog_manager.cc:4830] TS ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429): tablet bee6e9222daa40e09dbf75c0ffaf479d (table rebalance_test_table_4 [id=9a0826219eb944058c62c4f29729bf0e]) successfully deleted
I20240503 08:11:43.355249 32574 consensus_queue.cc:237] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } }
I20240503 08:11:43.356210 454 raft_consensus.cc:2932] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } } }
I20240503 08:11:43.359721 30653 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2b5f4bb2e73c4643a964b3c834b820aa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:11:43.361277 30667 catalog_manager.cc:5484] T 2b5f4bb2e73c4643a964b3c834b820aa P 03e83fbce8ee4c5cacbf54f25e1f2665 reported cstate change: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New cstate: current_term: 2 leader_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "03e83fbce8ee4c5cacbf54f25e1f2665" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 36035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:43.365407 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b5f4bb2e73c4643a964b3c834b820aa with delete_type TABLET_DATA_TOMBSTONED (TS 3986faa56d004ea5b787a469ddb79c81 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53736
I20240503 08:11:43.365810 741 tablet_replica.cc:330] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: stopping tablet replica
I20240503 08:11:43.366029 741 raft_consensus.cc:2229] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:43.366257 741 raft_consensus.cc:2258] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:43.367154 741 ts_tablet_manager.cc:1875] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:43.368191 32041 consensus_queue.cc:237] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } }
I20240503 08:11:43.369350 449 raft_consensus.cc:2932] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } } }
I20240503 08:11:43.372937 741 ts_tablet_manager.cc:1888] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:43.373064 741 log.cc:1196] T 2b5f4bb2e73c4643a964b3c834b820aa P 3986faa56d004ea5b787a469ddb79c81: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/2b5f4bb2e73c4643a964b3c834b820aa
I20240503 08:11:43.373827 30654 catalog_manager.cc:4830] TS 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943): tablet 2b5f4bb2e73c4643a964b3c834b820aa (table rebalance_test_table_3 [id=4074a9f3638e495cb809de913789b611]) successfully deleted
I20240503 08:11:43.374557 30656 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b123f5c42af4d11b4325b3cd6226c29 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:11:43.376708 30667 catalog_manager.cc:5484] T 6b123f5c42af4d11b4325b3cd6226c29 P a9b83be26ed14471b0335ffb28f12193 reported cstate change: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New cstate: current_term: 2 leader_uuid: "a9b83be26ed14471b0335ffb28f12193" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a9b83be26ed14471b0335ffb28f12193" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 42421 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:43.380527 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b123f5c42af4d11b4325b3cd6226c29 with delete_type TABLET_DATA_TOMBSTONED (TS 3986faa56d004ea5b787a469ddb79c81 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53736
I20240503 08:11:43.381055 741 tablet_replica.cc:330] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: stopping tablet replica
I20240503 08:11:43.381242 741 raft_consensus.cc:2229] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:43.381461 741 raft_consensus.cc:2258] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:43.382515 741 ts_tablet_manager.cc:1875] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:43.388856 741 ts_tablet_manager.cc:1888] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:43.389024 741 log.cc:1196] T 6b123f5c42af4d11b4325b3cd6226c29 P 3986faa56d004ea5b787a469ddb79c81: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/6b123f5c42af4d11b4325b3cd6226c29
I20240503 08:11:43.390014 30654 catalog_manager.cc:4830] TS 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943): tablet 6b123f5c42af4d11b4325b3cd6226c29 (table rebalance_test_table_2 [id=48d5c85892af46148fda788532045ac1]) successfully deleted
I20240503 08:11:43.549968 32554 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:43.561305 32424 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:43.564746 32152 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:43.587106 30790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:43.596773 32021 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:43.609771 30921 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:43.644066 32283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:43.810684 32554 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:43.824398 30790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:43.835024 32152 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:43.838063 32021 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:43.838616 32424 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:43.853812 30921 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:43.876214 32283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:44.593573 32554 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:44.594561 32424 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:44.635459 30790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:44.637241 32021 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:44.647711 32283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:44.656538 32152 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:44.665876 30921 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:44.876591 32554 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:44.887953 32424 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:44.899467 32152 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:44.910719 30790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:44.946432 32021 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:44.961066 32283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:44.961799 30921 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:45.254505 32554 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:45.269944 30790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:45.273777 32424 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:45.279036 32021 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:45.291775 32152 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:45.301564 30921 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:45.309265 32283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
d2cdaf8faf8c4440943fe8633faf60f7 | 127.27.157.254:36149 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.27.157.212:44489 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
03e83fbce8ee4c5cacbf54f25e1f2665 | 127.27.157.199:36035 | HEALTHY | <none> | 2 | 0
3986faa56d004ea5b787a469ddb79c81 | 127.27.157.193:44943 | HEALTHY | <none> | 3 | 0
9f45625e8fff471382d1376ff8cf0d00 | 127.27.157.198:42339 | HEALTHY | <none> | 2 | 0
a7b227b516a24a1b837415111eba1303 | 127.27.157.196:42725 | HEALTHY | <none> | 2 | 0
a9b83be26ed14471b0335ffb28f12193 | 127.27.157.195:42421 | HEALTHY | <none> | 2 | 0
ac3512789abc4bd890102fa6910f980b | 127.27.157.194:36429 | HEALTHY | <none> | 3 | 0
c5dc2c8f44d44889b3888f6265d07de9 | 127.27.157.197:34939 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.27.157.193 | experimental | 127.27.157.193:44943
local_ip_for_outbound_sockets | 127.27.157.194 | experimental | 127.27.157.194:36429
local_ip_for_outbound_sockets | 127.27.157.195 | experimental | 127.27.157.195:42421
local_ip_for_outbound_sockets | 127.27.157.196 | experimental | 127.27.157.196:42725
local_ip_for_outbound_sockets | 127.27.157.197 | experimental | 127.27.157.197:34939
local_ip_for_outbound_sockets | 127.27.157.198 | experimental | 127.27.157.198:42339
local_ip_for_outbound_sockets | 127.27.157.199 | experimental | 127.27.157.199:36035
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.27.157.193:44943
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.27.157.194:36429
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.27.157.195:42421
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.27.157.196:42725
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.27.157.197:34939
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.27.157.198:42339
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.27.157.199:36035
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.27.157.212:44489 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:11:45.387272 28279 log_verifier.cc:126] Checking tablet 153c5b78de3342eab81fdce1a7f199d5
I20240503 08:11:45.404337 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 153c5b78de3342eab81fdce1a7f199d5
I20240503 08:11:45.404512 28279 log_verifier.cc:126] Checking tablet 1d0ad95a564241cf8a0e5c674e920a0b
I20240503 08:11:45.416903 28279 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1d0ad95a564241cf8a0e5c674e920a0b
I20240503 08:11:45.417028 28279 log_verifier.cc:126] Checking tablet 1f4d7f23456f4bddabcdd74ddea40a33
I20240503 08:11:45.430119 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1f4d7f23456f4bddabcdd74ddea40a33
I20240503 08:11:45.430259 28279 log_verifier.cc:126] Checking tablet 20b5e4e7b78f4f0d979b936d92dd1eb3
I20240503 08:11:45.449501 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 20b5e4e7b78f4f0d979b936d92dd1eb3
I20240503 08:11:45.449662 28279 log_verifier.cc:126] Checking tablet 2b5f4bb2e73c4643a964b3c834b820aa
I20240503 08:11:45.463838 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2b5f4bb2e73c4643a964b3c834b820aa
I20240503 08:11:45.463966 28279 log_verifier.cc:126] Checking tablet 35b0575c97db429fb6c30261552e090b
I20240503 08:11:45.477006 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 35b0575c97db429fb6c30261552e090b
I20240503 08:11:45.477126 28279 log_verifier.cc:126] Checking tablet 38d8172f2dd1427fb7123752835bd458
I20240503 08:11:45.489792 28279 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 38d8172f2dd1427fb7123752835bd458
I20240503 08:11:45.489907 28279 log_verifier.cc:126] Checking tablet 64aacda851904a08b428c85b2842e126
I20240503 08:11:45.502287 28279 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 64aacda851904a08b428c85b2842e126
I20240503 08:11:45.502414 28279 log_verifier.cc:126] Checking tablet 6b123f5c42af4d11b4325b3cd6226c29
I20240503 08:11:45.516045 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6b123f5c42af4d11b4325b3cd6226c29
I20240503 08:11:45.516175 28279 log_verifier.cc:126] Checking tablet 6e188dd3547440b183799c9a796b4a85
I20240503 08:11:45.529358 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6e188dd3547440b183799c9a796b4a85
I20240503 08:11:45.529489 28279 log_verifier.cc:126] Checking tablet 74ae8557f32444f6b60487f75b6e353a
I20240503 08:11:45.541604 28279 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 74ae8557f32444f6b60487f75b6e353a
I20240503 08:11:45.541736 28279 log_verifier.cc:126] Checking tablet 87f2891ba60d452483f1edaf2a7f9eb3
I20240503 08:11:45.554545 28279 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 87f2891ba60d452483f1edaf2a7f9eb3
I20240503 08:11:45.554685 28279 log_verifier.cc:126] Checking tablet bee6e9222daa40e09dbf75c0ffaf479d
I20240503 08:11:45.567957 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet bee6e9222daa40e09dbf75c0ffaf479d
I20240503 08:11:45.568095 28279 log_verifier.cc:126] Checking tablet ded8fc9e27bf4613bf2f3dd4e9db250a
I20240503 08:11:45.576155 28279 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ded8fc9e27bf4613bf2f3dd4e9db250a
I20240503 08:11:45.576290 28279 log_verifier.cc:126] Checking tablet efe80a3ea5154c3a86d3816ba79657ce
I20240503 08:11:45.589085 28279 log_verifier.cc:177] Verified matching terms for 3 ops in tablet efe80a3ea5154c3a86d3816ba79657ce
I20240503 08:11:45.589215 28279 log_verifier.cc:126] Checking tablet fa28377977cf4ffa9347b5d45020c377
I20240503 08:11:45.602514 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fa28377977cf4ffa9347b5d45020c377
I20240503 08:11:45.605855 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.200:0
--local_ip_for_outbound_sockets=127.27.157.200
--webserver_interface=127.27.157.200
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--builtin_ntp_servers=127.27.157.212:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:45.985049 1070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:45.985392 1070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:45.985550 1070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:45.993366 1070 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:45.993532 1070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:45.993685 1070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.200
I20240503 08:11:46.003473 1070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.157.200
--webserver_port=0
--tserver_master_addrs=127.27.157.254:36149
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.200
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:46.004865 1070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:46.006654 1070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:46.020920 1075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:46.021788 1078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:46.022666 1076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:46.023684 1070 server_base.cc:1034] running on GCE node
I20240503 08:11:46.024363 1070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:46.025463 1070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:46.026716 1070 hybrid_clock.cc:648] HybridClock initialized: now 1714723906026652 us; error 60 us; skew 500 ppm
I20240503 08:11:46.027063 1070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:46.029928 1070 webserver.cc:460] Webserver started at http://127.27.157.200:36813/ using document root <none> and password file <none>
I20240503 08:11:46.030782 1070 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:46.030905 1070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:46.031234 1070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:46.034161 1070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "6d4b805d0eed446da6c46c8359ca0673"
format_stamp: "Formatted at 2024-05-03 08:11:46 on dist-test-slave-dj8w"
I20240503 08:11:46.035017 1070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "6d4b805d0eed446da6c46c8359ca0673"
format_stamp: "Formatted at 2024-05-03 08:11:46 on dist-test-slave-dj8w"
I20240503 08:11:46.040856 1070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 08:11:46.044911 1084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:46.046470 1070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:11:46.046650 1070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6d4b805d0eed446da6c46c8359ca0673"
format_stamp: "Formatted at 2024-05-03 08:11:46 on dist-test-slave-dj8w"
I20240503 08:11:46.046864 1070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:46.077196 1070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:46.078279 1070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:46.078610 1070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:46.079908 1070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:46.082089 1070 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:46.082208 1070 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:46.082373 1070 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:46.082468 1070 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:46.125226 1070 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.200:45771
I20240503 08:11:46.125311 1196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.200:45771 every 8 connection(s)
I20240503 08:11:46.126678 1070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:11:46.128093 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 1070
I20240503 08:11:46.128443 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:11:46.143997 1197 heartbeater.cc:347] Connected to a master server at 127.27.157.254:36149
I20240503 08:11:46.144392 1197 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:46.145337 1197 heartbeater.cc:511] Master 127.27.157.254:36149 requested a full tablet report, sending...
I20240503 08:11:46.146886 30660 ts_manager.cc:194] Registered new tserver with Master: 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200:45771)
I20240503 08:11:46.147759 30660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.200:58847
I20240503 08:11:46.649689 32554 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.669463 30790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:46.678510 32424 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.686504 1132 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:46.707427 32021 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.711905 32152 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:46.747810 30921 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:46.758977 32283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.034533 32554 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.053768 1132 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:11:47.056102 30790 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:47.073290 32152 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.077646 32424 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.143354 32021 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.144582 30921 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:11:47.150511 1197 heartbeater.cc:503] Master 127.27.157.254:36149 was elected leader, sending a full tablet report...
I20240503 08:11:47.173635 32283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:47.230283 30810 consensus_queue.cc:237] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: true } }
I20240503 08:11:47.232287 1289 raft_consensus.cc:2932] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: true } } }
I20240503 08:11:47.238382 30667 catalog_manager.cc:5484] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index -1 to 4, NON_VOTER 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:47.242787 30746 consensus_peers.cc:485] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 -> Peer 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200:45771): Couldn't send request to peer 6d4b805d0eed446da6c46c8359ca0673. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38d8172f2dd1427fb7123752835bd458. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:47.251408 30941 consensus_queue.cc:237] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: true } }
I20240503 08:11:47.253423 1302 raft_consensus.cc:2932] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: true } } }
I20240503 08:11:47.259266 30667 catalog_manager.cc:5484] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b reported cstate change: config changed from index -1 to 4, NON_VOTER 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:11:47.263777 30877 consensus_peers.cc:485] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b -> Peer 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200:45771): Couldn't send request to peer 6d4b805d0eed446da6c46c8359ca0673. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64aacda851904a08b428c85b2842e126. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:47.778095 1336 ts_tablet_manager.cc:902] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673: Initiating tablet copy from peer ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429)
I20240503 08:11:47.779918 1336 tablet_copy_client.cc:308] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:36429
I20240503 08:11:47.786881 30961 tablet_copy_service.cc:140] P ac3512789abc4bd890102fa6910f980b: Received BeginTabletCopySession request for tablet 64aacda851904a08b428c85b2842e126 from peer 6d4b805d0eed446da6c46c8359ca0673 ({username='slave'} at 127.27.157.200:60305)
I20240503 08:11:47.787058 30961 tablet_copy_service.cc:161] P ac3512789abc4bd890102fa6910f980b: Beginning new tablet copy session on tablet 64aacda851904a08b428c85b2842e126 from peer 6d4b805d0eed446da6c46c8359ca0673 at {username='slave'} at 127.27.157.200:60305: session id = 6d4b805d0eed446da6c46c8359ca0673-64aacda851904a08b428c85b2842e126
I20240503 08:11:47.789192 30961 tablet_copy_source_session.cc:205] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:47.793699 1336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64aacda851904a08b428c85b2842e126. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:47.803707 1336 tablet_copy_client.cc:783] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:47.804159 1336 tablet_copy_client.cc:647] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:47.806669 1336 tablet_copy_client.cc:523] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:47.810158 1336 tablet_bootstrap.cc:492] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673: Bootstrap starting.
I20240503 08:11:47.820159 1336 log.cc:824] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:47.838016 1344 ts_tablet_manager.cc:902] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673: Initiating tablet copy from peer 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:47.839802 1344 tablet_copy_client.cc:308] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:44943
I20240503 08:11:47.846666 30830 tablet_copy_service.cc:140] P 3986faa56d004ea5b787a469ddb79c81: Received BeginTabletCopySession request for tablet 38d8172f2dd1427fb7123752835bd458 from peer 6d4b805d0eed446da6c46c8359ca0673 ({username='slave'} at 127.27.157.200:48403)
I20240503 08:11:47.846930 30830 tablet_copy_service.cc:161] P 3986faa56d004ea5b787a469ddb79c81: Beginning new tablet copy session on tablet 38d8172f2dd1427fb7123752835bd458 from peer 6d4b805d0eed446da6c46c8359ca0673 at {username='slave'} at 127.27.157.200:48403: session id = 6d4b805d0eed446da6c46c8359ca0673-38d8172f2dd1427fb7123752835bd458
I20240503 08:11:47.850028 30830 tablet_copy_source_session.cc:205] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:11:47.851867 1344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d8172f2dd1427fb7123752835bd458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:47.859098 1344 tablet_copy_client.cc:783] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:47.859548 1344 tablet_copy_client.cc:647] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673: tablet copy: Starting download of 1 WAL segments...
I20240503 08:11:47.861915 1344 tablet_copy_client.cc:523] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:47.866104 1344 tablet_bootstrap.cc:492] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673: Bootstrap starting.
I20240503 08:11:47.867870 1336 tablet_bootstrap.cc:492] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:47.868542 1336 tablet_bootstrap.cc:492] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673: Bootstrap complete.
I20240503 08:11:47.869088 1336 ts_tablet_manager.cc:1367] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673: Time spent bootstrapping tablet: real 0.059s user 0.051s sys 0.009s
I20240503 08:11:47.875247 1336 raft_consensus.cc:348] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: true } }
I20240503 08:11:47.875777 1336 raft_consensus.cc:729] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d4b805d0eed446da6c46c8359ca0673, State: Initialized, Role: LEARNER
I20240503 08:11:47.876349 1336 consensus_queue.cc:260] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: true } }
I20240503 08:11:47.878245 1336 ts_tablet_manager.cc:1398] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240503 08:11:47.879303 30961 tablet_copy_service.cc:342] P ac3512789abc4bd890102fa6910f980b: Request end of tablet copy session 6d4b805d0eed446da6c46c8359ca0673-64aacda851904a08b428c85b2842e126 received from {username='slave'} at 127.27.157.200:60305
I20240503 08:11:47.879518 30961 tablet_copy_service.cc:433] P ac3512789abc4bd890102fa6910f980b: ending tablet copy session 6d4b805d0eed446da6c46c8359ca0673-64aacda851904a08b428c85b2842e126 on tablet 64aacda851904a08b428c85b2842e126 with peer 6d4b805d0eed446da6c46c8359ca0673
I20240503 08:11:47.902196 1344 tablet_bootstrap.cc:492] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:47.902619 1344 tablet_bootstrap.cc:492] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673: Bootstrap complete.
I20240503 08:11:47.902899 1344 ts_tablet_manager.cc:1367] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673: Time spent bootstrapping tablet: real 0.037s user 0.030s sys 0.004s
I20240503 08:11:47.903829 1344 raft_consensus.cc:348] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: true } }
I20240503 08:11:47.904028 1344 raft_consensus.cc:729] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d4b805d0eed446da6c46c8359ca0673, State: Initialized, Role: LEARNER
I20240503 08:11:47.904211 1344 consensus_queue.cc:260] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: true } }
I20240503 08:11:47.904692 1344 ts_tablet_manager.cc:1398] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:11:47.905652 30830 tablet_copy_service.cc:342] P 3986faa56d004ea5b787a469ddb79c81: Request end of tablet copy session 6d4b805d0eed446da6c46c8359ca0673-38d8172f2dd1427fb7123752835bd458 received from {username='slave'} at 127.27.157.200:48403
I20240503 08:11:47.905833 30830 tablet_copy_service.cc:433] P 3986faa56d004ea5b787a469ddb79c81: ending tablet copy session 6d4b805d0eed446da6c46c8359ca0673-38d8172f2dd1427fb7123752835bd458 on tablet 38d8172f2dd1427fb7123752835bd458 with peer 6d4b805d0eed446da6c46c8359ca0673
I20240503 08:11:48.194885 1152 raft_consensus.cc:1203] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:48.366007 1152 raft_consensus.cc:1203] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:11:48.719664 1396 raft_consensus.cc:1050] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: attempting to promote NON_VOTER 6d4b805d0eed446da6c46c8359ca0673 to VOTER
I20240503 08:11:48.720244 1396 consensus_queue.cc:237] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } }
I20240503 08:11:48.722213 1152 raft_consensus.cc:1261] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 LEARNER]: Refusing update from remote peer ac3512789abc4bd890102fa6910f980b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:48.722852 1396 consensus_queue.cc:1035] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:48.726469 1397 raft_consensus.cc:2932] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } } }
I20240503 08:11:48.727232 1152 raft_consensus.cc:2932] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } } }
I20240503 08:11:48.731377 30668 catalog_manager.cc:5484] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b reported cstate change: config changed from index 4 to 5, 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac3512789abc4bd890102fa6910f980b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:48.831645 30941 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "64aacda851904a08b428c85b2842e126"
dest_uuid: "ac3512789abc4bd890102fa6910f980b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52822
I20240503 08:11:48.831902 30941 raft_consensus.cc:595] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:48.978395 1421 raft_consensus.cc:1050] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: attempting to promote NON_VOTER 6d4b805d0eed446da6c46c8359ca0673 to VOTER
I20240503 08:11:48.979049 1421 consensus_queue.cc:237] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } }
I20240503 08:11:48.980870 1152 raft_consensus.cc:1261] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 LEARNER]: Refusing update from remote peer 3986faa56d004ea5b787a469ddb79c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:11:48.981451 1421 consensus_queue.cc:1035] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:11:48.985131 1420 raft_consensus.cc:2932] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } } }
I20240503 08:11:48.986517 1152 raft_consensus.cc:2932] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } } }
I20240503 08:11:48.990518 30667 catalog_manager.cc:5484] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 reported cstate change: config changed from index 4 to 5, 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3986faa56d004ea5b787a469ddb79c81" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:49.061177 30941 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "64aacda851904a08b428c85b2842e126"
dest_uuid: "ac3512789abc4bd890102fa6910f980b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52838
I20240503 08:11:49.061427 30941 raft_consensus.cc:595] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:49.080574 30810 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38d8172f2dd1427fb7123752835bd458"
dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38408
I20240503 08:11:49.080813 30810 raft_consensus.cc:595] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:49.247268 1446 raft_consensus.cc:979] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: : Instructing follower 6d4b805d0eed446da6c46c8359ca0673 to start an election
I20240503 08:11:49.247474 1446 raft_consensus.cc:1067] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Signalling peer 6d4b805d0eed446da6c46c8359ca0673 to start an election
I20240503 08:11:49.248354 1152 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64aacda851904a08b428c85b2842e126"
dest_uuid: "6d4b805d0eed446da6c46c8359ca0673"
from {username='slave'} at 127.27.157.194:59219
I20240503 08:11:49.248673 1152 raft_consensus.cc:482] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:49.248873 1152 raft_consensus.cc:3037] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:49.250890 1152 raft_consensus.cc:504] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } }
I20240503 08:11:49.252321 1152 leader_election.cc:288] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [CANDIDATE]: Term 2 election: Requested vote from peers ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429)
I20240503 08:11:49.252967 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64aacda851904a08b428c85b2842e126" candidate_uuid: "6d4b805d0eed446da6c46c8359ca0673" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ac3512789abc4bd890102fa6910f980b"
I20240503 08:11:49.253211 30941 raft_consensus.cc:3032] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:49.253293 30941 raft_consensus.cc:729] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac3512789abc4bd890102fa6910f980b, State: Running, Role: LEADER
I20240503 08:11:49.253564 30941 consensus_queue.cc:260] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } }
I20240503 08:11:49.253822 30941 raft_consensus.cc:3037] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:49.255977 30941 raft_consensus.cc:2454] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4b805d0eed446da6c46c8359ca0673 in term 2.
I20240503 08:11:49.256716 1088 leader_election.cc:302] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6d4b805d0eed446da6c46c8359ca0673, ac3512789abc4bd890102fa6910f980b; no voters:
I20240503 08:11:49.257169 1350 raft_consensus.cc:2781] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:49.257587 1350 raft_consensus.cc:686] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4b805d0eed446da6c46c8359ca0673, State: Running, Role: LEADER
I20240503 08:11:49.258050 1350 consensus_queue.cc:237] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } }
I20240503 08:11:49.261545 30668 catalog_manager.cc:5484] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 reported cstate change: term changed from 1 to 2, leader changed from ac3512789abc4bd890102fa6910f980b (127.27.157.194) to 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200). New cstate: current_term: 2 leader_uuid: "6d4b805d0eed446da6c46c8359ca0673" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:11:49.271957 30656 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 64aacda851904a08b428c85b2842e126 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200:45771) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 08:11:49.308815 30810 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38d8172f2dd1427fb7123752835bd458"
dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38436
I20240503 08:11:49.309065 30810 raft_consensus.cc:595] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:11:49.483268 1424 raft_consensus.cc:979] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: : Instructing follower 6d4b805d0eed446da6c46c8359ca0673 to start an election
I20240503 08:11:49.483523 1421 raft_consensus.cc:1067] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Signalling peer 6d4b805d0eed446da6c46c8359ca0673 to start an election
I20240503 08:11:49.484400 1152 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38d8172f2dd1427fb7123752835bd458"
dest_uuid: "6d4b805d0eed446da6c46c8359ca0673"
from {username='slave'} at 127.27.157.193:42449
I20240503 08:11:49.484643 1152 raft_consensus.cc:482] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:49.484786 1152 raft_consensus.cc:3037] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:49.486806 1152 raft_consensus.cc:504] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } }
I20240503 08:11:49.487505 1152 leader_election.cc:288] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [CANDIDATE]: Term 2 election: Requested vote from peers 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943)
I20240503 08:11:49.488176 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d8172f2dd1427fb7123752835bd458" candidate_uuid: "6d4b805d0eed446da6c46c8359ca0673" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3986faa56d004ea5b787a469ddb79c81"
I20240503 08:11:49.488438 30810 raft_consensus.cc:3032] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:49.488529 30810 raft_consensus.cc:729] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3986faa56d004ea5b787a469ddb79c81, State: Running, Role: LEADER
I20240503 08:11:49.488888 30810 consensus_queue.cc:260] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } }
I20240503 08:11:49.489296 30810 raft_consensus.cc:3037] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:49.492275 30810 raft_consensus.cc:2454] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4b805d0eed446da6c46c8359ca0673 in term 2.
I20240503 08:11:49.492954 1086 leader_election.cc:302] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3986faa56d004ea5b787a469ddb79c81, 6d4b805d0eed446da6c46c8359ca0673; no voters:
I20240503 08:11:49.493340 1350 raft_consensus.cc:2781] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:49.493530 1350 raft_consensus.cc:686] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4b805d0eed446da6c46c8359ca0673, State: Running, Role: LEADER
I20240503 08:11:49.493856 1350 consensus_queue.cc:237] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } }
I20240503 08:11:49.496711 30668 catalog_manager.cc:5484] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 reported cstate change: term changed from 1 to 2, leader changed from 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) to 6d4b805d0eed446da6c46c8359ca0673 (127.27.157.200). New cstate: current_term: 2 leader_uuid: "6d4b805d0eed446da6c46c8359ca0673" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:49.878365 30941 raft_consensus.cc:1261] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Refusing update from remote peer 6d4b805d0eed446da6c46c8359ca0673: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:49.879155 1447 consensus_queue.cc:1035] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3512789abc4bd890102fa6910f980b" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 36429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:49.915411 1152 consensus_queue.cc:237] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } }
I20240503 08:11:49.916352 1447 raft_consensus.cc:2932] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ac3512789abc4bd890102fa6910f980b (127.27.157.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } } }
I20240503 08:11:49.919418 30656 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64aacda851904a08b428c85b2842e126 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:11:49.920779 30667 catalog_manager.cc:5484] T 64aacda851904a08b428c85b2842e126 P 6d4b805d0eed446da6c46c8359ca0673 reported cstate change: config changed from index 5 to 7, VOTER ac3512789abc4bd890102fa6910f980b (127.27.157.194) evicted. New cstate: current_term: 2 leader_uuid: "6d4b805d0eed446da6c46c8359ca0673" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:49.924587 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 64aacda851904a08b428c85b2842e126 with delete_type TABLET_DATA_TOMBSTONED (TS ac3512789abc4bd890102fa6910f980b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59376
I20240503 08:11:49.925257 1489 tablet_replica.cc:330] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: stopping tablet replica
I20240503 08:11:49.925439 1489 raft_consensus.cc:2229] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:49.925637 1489 raft_consensus.cc:2258] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:49.927495 1489 ts_tablet_manager.cc:1875] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:49.933847 1489 ts_tablet_manager.cc:1888] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:49.934023 1489 log.cc:1196] T 64aacda851904a08b428c85b2842e126 P ac3512789abc4bd890102fa6910f980b: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/64aacda851904a08b428c85b2842e126
I20240503 08:11:49.934834 30656 catalog_manager.cc:4830] TS ac3512789abc4bd890102fa6910f980b (127.27.157.194:36429): tablet 64aacda851904a08b428c85b2842e126 (table rebalance_test_table_2 [id=48d5c85892af46148fda788532045ac1]) successfully deleted
I20240503 08:11:49.981575 30810 raft_consensus.cc:1261] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Refusing update from remote peer 6d4b805d0eed446da6c46c8359ca0673: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:11:49.982244 1447 consensus_queue.cc:1035] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3986faa56d004ea5b787a469ddb79c81" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 44943 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:11:50.115832 1152 consensus_queue.cc:237] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } }
I20240503 08:11:50.116719 1447 raft_consensus.cc:2932] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } } }
I20240503 08:11:50.119529 30656 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 38d8172f2dd1427fb7123752835bd458 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 08:11:50.120852 30668 catalog_manager.cc:5484] T 38d8172f2dd1427fb7123752835bd458 P 6d4b805d0eed446da6c46c8359ca0673 reported cstate change: config changed from index 5 to 7, VOTER 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193) evicted. New cstate: current_term: 2 leader_uuid: "6d4b805d0eed446da6c46c8359ca0673" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d4b805d0eed446da6c46c8359ca0673" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45771 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:11:50.124459 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d8172f2dd1427fb7123752835bd458 with delete_type TABLET_DATA_TOMBSTONED (TS 3986faa56d004ea5b787a469ddb79c81 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53736
I20240503 08:11:50.125229 1506 tablet_replica.cc:330] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: stopping tablet replica
I20240503 08:11:50.125470 1506 raft_consensus.cc:2229] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:50.125732 1506 raft_consensus.cc:2258] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:50.127733 1506 ts_tablet_manager.cc:1875] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:50.134755 1506 ts_tablet_manager.cc:1888] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:11:50.134914 1506 log.cc:1196] T 38d8172f2dd1427fb7123752835bd458 P 3986faa56d004ea5b787a469ddb79c81: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/38d8172f2dd1427fb7123752835bd458
I20240503 08:11:50.136036 30654 catalog_manager.cc:4830] TS 3986faa56d004ea5b787a469ddb79c81 (127.27.157.193:44943): tablet 38d8172f2dd1427fb7123752835bd458 (table rebalance_test_table_4 [id=9a0826219eb944058c62c4f29729bf0e]) successfully deleted
I20240503 08:11:50.294602 1132 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.304350 32554 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.325145 32424 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.343899 30921 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.343932 30790 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.375216 32152 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.385362 32021 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.395857 32283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.557127 30790 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.557929 1132 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.566016 32554 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.567487 32424 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.588701 32152 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.601770 32021 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.628996 30921 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.667553 32283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.922896 32554 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.931723 30790 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.934408 32424 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.941220 1132 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.975739 32021 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.975900 32152 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.981421 30921 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:11:50.982434 32283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
d2cdaf8faf8c4440943fe8633faf60f7 | 127.27.157.254:36149 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.27.157.212:44489 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
03e83fbce8ee4c5cacbf54f25e1f2665 | 127.27.157.199:36035 | HEALTHY | <none> | 2 | 0
3986faa56d004ea5b787a469ddb79c81 | 127.27.157.193:44943 | HEALTHY | <none> | 2 | 0
6d4b805d0eed446da6c46c8359ca0673 | 127.27.157.200:45771 | HEALTHY | <none> | 2 | 0
9f45625e8fff471382d1376ff8cf0d00 | 127.27.157.198:42339 | HEALTHY | <none> | 2 | 0
a7b227b516a24a1b837415111eba1303 | 127.27.157.196:42725 | HEALTHY | <none> | 2 | 0
a9b83be26ed14471b0335ffb28f12193 | 127.27.157.195:42421 | HEALTHY | <none> | 2 | 0
ac3512789abc4bd890102fa6910f980b | 127.27.157.194:36429 | HEALTHY | <none> | 2 | 0
c5dc2c8f44d44889b3888f6265d07de9 | 127.27.157.197:34939 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.27.157.193 | experimental | 127.27.157.193:44943
local_ip_for_outbound_sockets | 127.27.157.194 | experimental | 127.27.157.194:36429
local_ip_for_outbound_sockets | 127.27.157.195 | experimental | 127.27.157.195:42421
local_ip_for_outbound_sockets | 127.27.157.196 | experimental | 127.27.157.196:42725
local_ip_for_outbound_sockets | 127.27.157.197 | experimental | 127.27.157.197:34939
local_ip_for_outbound_sockets | 127.27.157.198 | experimental | 127.27.157.198:42339
local_ip_for_outbound_sockets | 127.27.157.199 | experimental | 127.27.157.199:36035
local_ip_for_outbound_sockets | 127.27.157.200 | experimental | 127.27.157.200:45771
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.27.157.193:44943
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.27.157.194:36429
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.27.157.195:42421
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.27.157.196:42725
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.27.157.197:34939
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.27.157.198:42339
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.27.157.199:36035
server_dump_info_path | /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.27.157.200:45771
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.27.157.212:44489 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 08:11:51.063066 28279 log_verifier.cc:126] Checking tablet 153c5b78de3342eab81fdce1a7f199d5
I20240503 08:11:51.073868 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 153c5b78de3342eab81fdce1a7f199d5
I20240503 08:11:51.073985 28279 log_verifier.cc:126] Checking tablet 1d0ad95a564241cf8a0e5c674e920a0b
I20240503 08:11:51.083825 28279 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1d0ad95a564241cf8a0e5c674e920a0b
I20240503 08:11:51.083918 28279 log_verifier.cc:126] Checking tablet 1f4d7f23456f4bddabcdd74ddea40a33
I20240503 08:11:51.094174 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1f4d7f23456f4bddabcdd74ddea40a33
I20240503 08:11:51.094290 28279 log_verifier.cc:126] Checking tablet 20b5e4e7b78f4f0d979b936d92dd1eb3
I20240503 08:11:51.105751 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 20b5e4e7b78f4f0d979b936d92dd1eb3
I20240503 08:11:51.105875 28279 log_verifier.cc:126] Checking tablet 2b5f4bb2e73c4643a964b3c834b820aa
I20240503 08:11:51.122043 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2b5f4bb2e73c4643a964b3c834b820aa
I20240503 08:11:51.122165 28279 log_verifier.cc:126] Checking tablet 35b0575c97db429fb6c30261552e090b
I20240503 08:11:51.133533 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 35b0575c97db429fb6c30261552e090b
I20240503 08:11:51.133632 28279 log_verifier.cc:126] Checking tablet 38d8172f2dd1427fb7123752835bd458
I20240503 08:11:51.146636 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 38d8172f2dd1427fb7123752835bd458
I20240503 08:11:51.146755 28279 log_verifier.cc:126] Checking tablet 64aacda851904a08b428c85b2842e126
I20240503 08:11:51.159385 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 64aacda851904a08b428c85b2842e126
I20240503 08:11:51.159523 28279 log_verifier.cc:126] Checking tablet 6b123f5c42af4d11b4325b3cd6226c29
I20240503 08:11:51.170269 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6b123f5c42af4d11b4325b3cd6226c29
I20240503 08:11:51.170372 28279 log_verifier.cc:126] Checking tablet 6e188dd3547440b183799c9a796b4a85
I20240503 08:11:51.180580 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6e188dd3547440b183799c9a796b4a85
I20240503 08:11:51.180686 28279 log_verifier.cc:126] Checking tablet 74ae8557f32444f6b60487f75b6e353a
I20240503 08:11:51.190379 28279 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 74ae8557f32444f6b60487f75b6e353a
I20240503 08:11:51.190505 28279 log_verifier.cc:126] Checking tablet 87f2891ba60d452483f1edaf2a7f9eb3
I20240503 08:11:51.200587 28279 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 87f2891ba60d452483f1edaf2a7f9eb3
I20240503 08:11:51.200690 28279 log_verifier.cc:126] Checking tablet bee6e9222daa40e09dbf75c0ffaf479d
I20240503 08:11:51.210731 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet bee6e9222daa40e09dbf75c0ffaf479d
I20240503 08:11:51.210855 28279 log_verifier.cc:126] Checking tablet ded8fc9e27bf4613bf2f3dd4e9db250a
I20240503 08:11:51.215929 28279 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ded8fc9e27bf4613bf2f3dd4e9db250a
I20240503 08:11:51.216029 28279 log_verifier.cc:126] Checking tablet efe80a3ea5154c3a86d3816ba79657ce
I20240503 08:11:51.225306 28279 log_verifier.cc:177] Verified matching terms for 3 ops in tablet efe80a3ea5154c3a86d3816ba79657ce
I20240503 08:11:51.225409 28279 log_verifier.cc:126] Checking tablet fa28377977cf4ffa9347b5d45020c377
I20240503 08:11:51.235855 28279 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fa28377977cf4ffa9347b5d45020c377
I20240503 08:11:51.236776 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 30728
I20240503 08:11:51.238250 30849 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:51.382083 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 30728
I20240503 08:11:51.408726 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 30859
I20240503 08:11:51.409819 30981 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:51.559032 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 30859
I20240503 08:11:51.584044 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 31959
I20240503 08:11:51.590616 32080 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:51.708245 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 31959
I20240503 08:11:51.729575 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 32090
I20240503 08:11:51.735889 32212 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:51.853080 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 32090
I20240503 08:11:51.874351 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 32220
I20240503 08:11:51.880734 32348 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:51.995743 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 32220
I20240503 08:11:52.016649 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 32358
I20240503 08:11:52.023006 32484 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:52.140117 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 32358
I20240503 08:11:52.157253 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 32492
I20240503 08:11:52.163897 32614 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:52.282035 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 32492
I20240503 08:11:52.303308 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 1070
I20240503 08:11:52.310024 1192 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:52.412597 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 1070
I20240503 08:11:52.431661 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 30638
I20240503 08:11:52.432929 30699 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:52.586285 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 30638
2024-05-03T08:11:52Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (28676 ms)
[----------] 1 test from RebalanceParamTest (28676 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240503 08:11:52.660737 28279 test_util.cc:268] Using random seed: 1169405672
I20240503 08:11:52.662674 28279 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:11:52.662743 28279 ts_itest-base.cc:113] --------------
I20240503 08:11:52.662788 28279 ts_itest-base.cc:114] 8 tablet servers
I20240503 08:11:52.662843 28279 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:11:52.662901 28279 ts_itest-base.cc:116] --------------
2024-05-03T08:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:11:52Z Disabled control of system clock
I20240503 08:11:52.694406 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.157.254:37795
--webserver_interface=127.27.157.254
--webserver_port=0
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.157.254:37795
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 08:11:53.037755 1641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:53.038089 1641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:53.038231 1641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:53.045957 1641 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:53.046103 1641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:11:53.046185 1641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:53.046269 1641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:11:53.046330 1641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:11:53.055912 1641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.157.254:37795
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.157.254:37795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.157.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:53.057319 1641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:53.059180 1641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:53.070996 1647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:53.071636 1646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:53.072427 1649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:53.073503 1641 server_base.cc:1034] running on GCE node
I20240503 08:11:53.074360 1641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:53.075870 1641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:53.077075 1641 hybrid_clock.cc:648] HybridClock initialized: now 1714723913077012 us; error 61 us; skew 500 ppm
I20240503 08:11:53.077422 1641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:53.079856 1641 webserver.cc:460] Webserver started at http://127.27.157.254:45051/ using document root <none> and password file <none>
I20240503 08:11:53.080610 1641 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:53.080739 1641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:53.081043 1641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:53.083823 1641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f97fddd2586f4c5cb4a95a0cbf96c0d0"
format_stamp: "Formatted at 2024-05-03 08:11:53 on dist-test-slave-dj8w"
I20240503 08:11:53.084625 1641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f97fddd2586f4c5cb4a95a0cbf96c0d0"
format_stamp: "Formatted at 2024-05-03 08:11:53 on dist-test-slave-dj8w"
I20240503 08:11:53.090075 1641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 08:11:53.094004 1655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:53.095626 1641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 08:11:53.095840 1641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f97fddd2586f4c5cb4a95a0cbf96c0d0"
format_stamp: "Formatted at 2024-05-03 08:11:53 on dist-test-slave-dj8w"
I20240503 08:11:53.096079 1641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:53.118021 1641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:53.118942 1641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:53.119244 1641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:53.138237 1641 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.254:37795
I20240503 08:11:53.138257 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.254:37795 every 8 connection(s)
I20240503 08:11:53.139684 1641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:11:53.143709 1707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:53.145882 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 1641
I20240503 08:11:53.146095 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:11:53.155561 1707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Bootstrap starting.
I20240503 08:11:53.158993 1707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:53.160204 1707 log.cc:824] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Log is configured to *not* fsync() on all Append() calls
I20240503 08:11:53.163090 1707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: No bootstrap required, opened a new log
I20240503 08:11:53.169592 1707 raft_consensus.cc:348] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97fddd2586f4c5cb4a95a0cbf96c0d0" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 37795 } }
I20240503 08:11:53.169988 1707 raft_consensus.cc:374] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:53.170102 1707 raft_consensus.cc:729] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f97fddd2586f4c5cb4a95a0cbf96c0d0, State: Initialized, Role: FOLLOWER
I20240503 08:11:53.170748 1707 consensus_queue.cc:260] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97fddd2586f4c5cb4a95a0cbf96c0d0" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 37795 } }
I20240503 08:11:53.170946 1707 raft_consensus.cc:388] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:53.171074 1707 raft_consensus.cc:482] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:53.171237 1707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:53.173550 1707 raft_consensus.cc:504] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97fddd2586f4c5cb4a95a0cbf96c0d0" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 37795 } }
I20240503 08:11:53.174103 1707 leader_election.cc:302] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f97fddd2586f4c5cb4a95a0cbf96c0d0; no voters:
I20240503 08:11:53.174702 1707 leader_election.cc:288] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:53.174870 1712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:53.175536 1712 raft_consensus.cc:686] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [term 1 LEADER]: Becoming Leader. State: Replica: f97fddd2586f4c5cb4a95a0cbf96c0d0, State: Running, Role: LEADER
I20240503 08:11:53.176075 1712 consensus_queue.cc:237] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97fddd2586f4c5cb4a95a0cbf96c0d0" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 37795 } }
I20240503 08:11:53.176689 1707 sys_catalog.cc:564] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:53.179562 1713 sys_catalog.cc:455] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f97fddd2586f4c5cb4a95a0cbf96c0d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97fddd2586f4c5cb4a95a0cbf96c0d0" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 37795 } } }
I20240503 08:11:53.179855 1714 sys_catalog.cc:455] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f97fddd2586f4c5cb4a95a0cbf96c0d0. Latest consensus state: current_term: 1 leader_uuid: "f97fddd2586f4c5cb4a95a0cbf96c0d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f97fddd2586f4c5cb4a95a0cbf96c0d0" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 37795 } } }
I20240503 08:11:53.180032 1713 sys_catalog.cc:458] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:53.180133 1714 sys_catalog.cc:458] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:53.181695 1718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:53.187000 1718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:53.196617 1718 catalog_manager.cc:1347] Generated new cluster ID: 0ba4a25a016c49189224c13dbc5deb99
I20240503 08:11:53.196753 1718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:53.223584 1718 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:53.224865 1718 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:53.236799 1718 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Generated new TSK 0
I20240503 08:11:53.237850 1718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:53.255031 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.193:0
--local_ip_for_outbound_sockets=127.27.157.193
--webserver_interface=127.27.157.193
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:11:53.596874 1731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:53.597168 1731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:53.597299 1731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:53.605110 1731 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:53.605244 1731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:53.605410 1731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.193
I20240503 08:11:53.614554 1731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.157.193
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.193
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:53.615962 1731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:53.617694 1731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:53.630209 1736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:53.630926 1737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:53.631600 1739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:53.632453 1731 server_base.cc:1034] running on GCE node
I20240503 08:11:53.633107 1731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:53.634106 1731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:53.635313 1731 hybrid_clock.cc:648] HybridClock initialized: now 1714723913635261 us; error 42 us; skew 500 ppm
I20240503 08:11:53.635679 1731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:53.642252 1731 webserver.cc:460] Webserver started at http://127.27.157.193:41447/ using document root <none> and password file <none>
I20240503 08:11:53.643067 1731 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:53.643195 1731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:53.643577 1731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:53.646342 1731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b8f581390da449ccb9e299011296405e"
format_stamp: "Formatted at 2024-05-03 08:11:53 on dist-test-slave-dj8w"
I20240503 08:11:53.647135 1731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b8f581390da449ccb9e299011296405e"
format_stamp: "Formatted at 2024-05-03 08:11:53 on dist-test-slave-dj8w"
I20240503 08:11:53.652563 1731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 08:11:53.656080 1745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:53.657575 1731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:53.657749 1731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b8f581390da449ccb9e299011296405e"
format_stamp: "Formatted at 2024-05-03 08:11:53 on dist-test-slave-dj8w"
I20240503 08:11:53.657955 1731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:53.682549 1731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:53.683478 1731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:53.683753 1731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:53.684733 1731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:53.686285 1731 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:53.686376 1731 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:53.686460 1731 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:53.686517 1731 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:53.714890 1731 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.193:41181
I20240503 08:11:53.714944 1857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.193:41181 every 8 connection(s)
I20240503 08:11:53.716301 1731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:11:53.716969 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 1731
I20240503 08:11:53.717245 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:11:53.721079 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.194:0
--local_ip_for_outbound_sockets=127.27.157.194
--webserver_interface=127.27.157.194
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:53.731642 1858 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:11:53.731967 1858 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:53.732856 1858 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:11:53.735306 1672 ts_manager.cc:194] Registered new tserver with Master: b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:11:53.737869 1672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.193:38615
W20240503 08:11:54.057442 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:54.057773 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:54.057870 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:54.065657 1862 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:54.065797 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:54.065914 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.194
I20240503 08:11:54.075232 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.157.194
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.194
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:54.076583 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:54.078397 1862 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:54.091030 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:54.091801 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:54.092677 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:54.093806 1862 server_base.cc:1034] running on GCE node
I20240503 08:11:54.094508 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:54.095603 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:54.096856 1862 hybrid_clock.cc:648] HybridClock initialized: now 1714723914096797 us; error 51 us; skew 500 ppm
I20240503 08:11:54.097195 1862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:54.099608 1862 webserver.cc:460] Webserver started at http://127.27.157.194:45497/ using document root <none> and password file <none>
I20240503 08:11:54.100421 1862 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:54.100548 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:54.100876 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:54.103718 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dj8w"
I20240503 08:11:54.104524 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dj8w"
I20240503 08:11:54.110302 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 08:11:54.113956 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:54.115276 1862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:11:54.115497 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dj8w"
I20240503 08:11:54.115736 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:54.140064 1862 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:54.141212 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:54.141566 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:54.142784 1862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:54.144881 1862 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:54.144994 1862 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:54.145099 1862 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:54.145227 1862 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:54.181281 1862 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.194:41975
I20240503 08:11:54.181332 1988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.194:41975 every 8 connection(s)
I20240503 08:11:54.182596 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:11:54.185045 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 1862
I20240503 08:11:54.185242 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:11:54.188766 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.195:0
--local_ip_for_outbound_sockets=127.27.157.195
--webserver_interface=127.27.157.195
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:54.196554 1989 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:11:54.196971 1989 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:54.198056 1989 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:11:54.199798 1672 ts_manager.cc:194] Registered new tserver with Master: 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:11:54.200973 1672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.194:56705
W20240503 08:11:54.532598 1992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:54.532920 1992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:54.533012 1992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:54.540705 1992 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:54.540851 1992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:54.540973 1992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.195
I20240503 08:11:54.550154 1992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.157.195
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.195
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:54.551491 1992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:54.553236 1992 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:54.565658 1997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:54.566601 1998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:54.567214 2000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:54.568202 1992 server_base.cc:1034] running on GCE node
I20240503 08:11:54.568840 1992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:54.569866 1992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:54.571065 1992 hybrid_clock.cc:648] HybridClock initialized: now 1714723914571011 us; error 56 us; skew 500 ppm
I20240503 08:11:54.571421 1992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:54.573743 1992 webserver.cc:460] Webserver started at http://127.27.157.195:39481/ using document root <none> and password file <none>
I20240503 08:11:54.574506 1992 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:54.574633 1992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:54.574965 1992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:54.577757 1992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "43e2e01cdd314f9c872c4336ef74250a"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dj8w"
I20240503 08:11:54.578567 1992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "43e2e01cdd314f9c872c4336ef74250a"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dj8w"
I20240503 08:11:54.583914 1992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 08:11:54.587479 2006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:54.588863 1992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:54.589021 1992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "43e2e01cdd314f9c872c4336ef74250a"
format_stamp: "Formatted at 2024-05-03 08:11:54 on dist-test-slave-dj8w"
I20240503 08:11:54.589226 1992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:54.606832 1992 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:54.607779 1992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:54.608071 1992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:54.609056 1992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:54.610600 1992 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:54.610702 1992 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:54.610827 1992 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:54.610899 1992 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:54.644534 1992 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.195:43117
I20240503 08:11:54.644593 2118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.195:43117 every 8 connection(s)
I20240503 08:11:54.645941 1992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:11:54.653098 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 1992
I20240503 08:11:54.653331 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:11:54.657342 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.196:0
--local_ip_for_outbound_sockets=127.27.157.196
--webserver_interface=127.27.157.196
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:54.664139 2119 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:11:54.664489 2119 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:54.665370 2119 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:11:54.666908 1672 ts_manager.cc:194] Registered new tserver with Master: 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:11:54.668164 1672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.195:42979
I20240503 08:11:54.741297 1858 heartbeater.cc:503] Master 127.27.157.254:37795 was elected leader, sending a full tablet report...
W20240503 08:11:55.005460 2122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:55.005787 2122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:55.005882 2122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:55.013568 2122 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:55.013726 2122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:55.013842 2122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.196
I20240503 08:11:55.023382 2122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.157.196
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.196
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:55.024727 2122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:55.026512 2122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:55.039777 2127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:55.040697 2128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:55.041411 2130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:55.042289 2122 server_base.cc:1034] running on GCE node
I20240503 08:11:55.042963 2122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:55.044066 2122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:55.045290 2122 hybrid_clock.cc:648] HybridClock initialized: now 1714723915045235 us; error 52 us; skew 500 ppm
I20240503 08:11:55.045642 2122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:55.048251 2122 webserver.cc:460] Webserver started at http://127.27.157.196:37499/ using document root <none> and password file <none>
I20240503 08:11:55.049129 2122 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:55.049268 2122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:55.049587 2122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:55.052448 2122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "184c543cb05d49cc98ba1c56617be7a5"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dj8w"
I20240503 08:11:55.053313 2122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "184c543cb05d49cc98ba1c56617be7a5"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dj8w"
I20240503 08:11:55.059109 2122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 08:11:55.063042 2136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:55.064635 2122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:11:55.064808 2122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "184c543cb05d49cc98ba1c56617be7a5"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dj8w"
I20240503 08:11:55.065018 2122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:55.091287 2122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:55.092263 2122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:55.092558 2122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:55.093578 2122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:55.095206 2122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:55.095309 2122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:55.095458 2122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:55.095551 2122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:55.129371 2122 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.196:42183
I20240503 08:11:55.129426 2248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.196:42183 every 8 connection(s)
I20240503 08:11:55.130862 2122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:11:55.132961 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 2122
I20240503 08:11:55.133175 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:11:55.136372 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.197:0
--local_ip_for_outbound_sockets=127.27.157.197
--webserver_interface=127.27.157.197
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:55.145292 2249 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:11:55.145622 2249 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:55.146492 2249 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:11:55.147951 1672 ts_manager.cc:194] Registered new tserver with Master: 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:11:55.148778 1672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.196:35221
I20240503 08:11:55.204466 1989 heartbeater.cc:503] Master 127.27.157.254:37795 was elected leader, sending a full tablet report...
W20240503 08:11:55.481066 2252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:55.481513 2252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:55.481684 2252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:55.489842 2252 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:55.489979 2252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:55.490129 2252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.197
I20240503 08:11:55.499727 2252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.157.197
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.197
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:55.501164 2252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:55.503007 2252 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:55.515666 2257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:55.516544 2258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:55.517107 2260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:55.518174 2252 server_base.cc:1034] running on GCE node
I20240503 08:11:55.518857 2252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:55.519917 2252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:55.521152 2252 hybrid_clock.cc:648] HybridClock initialized: now 1714723915521078 us; error 69 us; skew 500 ppm
I20240503 08:11:55.521557 2252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:55.524163 2252 webserver.cc:460] Webserver started at http://127.27.157.197:37373/ using document root <none> and password file <none>
I20240503 08:11:55.525025 2252 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:55.525169 2252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:55.525509 2252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:55.528477 2252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5f95b05d111e4090bbc59dec394ed0ab"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dj8w"
I20240503 08:11:55.529361 2252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5f95b05d111e4090bbc59dec394ed0ab"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dj8w"
I20240503 08:11:55.535269 2252 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240503 08:11:55.539235 2266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:55.540891 2252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:55.541090 2252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5f95b05d111e4090bbc59dec394ed0ab"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dj8w"
I20240503 08:11:55.541316 2252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:55.567374 2252 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:55.568331 2252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:55.568636 2252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:55.569671 2252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:55.571403 2252 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:55.571514 2252 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:55.571636 2252 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:55.571713 2252 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:55.606316 2252 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.197:42465
I20240503 08:11:55.606396 2378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.197:42465 every 8 connection(s)
I20240503 08:11:55.607808 2252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:11:55.613149 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 2252
I20240503 08:11:55.613400 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:11:55.617540 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.198:0
--local_ip_for_outbound_sockets=127.27.157.198
--webserver_interface=127.27.157.198
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:55.624349 2379 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:11:55.624703 2379 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:55.625622 2379 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:11:55.627370 1672 ts_manager.cc:194] Registered new tserver with Master: 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:11:55.628225 1672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.197:58103
I20240503 08:11:55.671572 2119 heartbeater.cc:503] Master 127.27.157.254:37795 was elected leader, sending a full tablet report...
W20240503 08:11:55.982712 2382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:55.983028 2382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:55.983151 2382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:55.991703 2382 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:55.991858 2382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:55.992054 2382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.198
I20240503 08:11:56.001652 2382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.157.198
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.198
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:56.003145 2382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:56.005028 2382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:56.017895 2387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.018571 2388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.019413 2390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:56.020293 2382 server_base.cc:1034] running on GCE node
I20240503 08:11:56.021030 2382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:56.022140 2382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:56.023422 2382 hybrid_clock.cc:648] HybridClock initialized: now 1714723916023362 us; error 42 us; skew 500 ppm
I20240503 08:11:56.023793 2382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:56.026353 2382 webserver.cc:460] Webserver started at http://127.27.157.198:42091/ using document root <none> and password file <none>
I20240503 08:11:56.027218 2382 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:56.027379 2382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:56.027745 2382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:56.030563 2382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "31f41542452b4b1eb994ff4223d039fc"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:11:56.031466 2382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "31f41542452b4b1eb994ff4223d039fc"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:11:56.037129 2382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240503 08:11:56.040861 2396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:56.042315 2382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:56.042510 2382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "31f41542452b4b1eb994ff4223d039fc"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:11:56.042734 2382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:56.059847 2382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:56.060736 2382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:56.061076 2382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:56.062103 2382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:56.063853 2382 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:56.063947 2382 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:56.064034 2382 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:56.064092 2382 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:56.097993 2382 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.198:46559
I20240503 08:11:56.098047 2508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.198:46559 every 8 connection(s)
I20240503 08:11:56.099455 2382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:11:56.103075 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 2382
I20240503 08:11:56.103269 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:11:56.107282 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.199:0
--local_ip_for_outbound_sockets=127.27.157.199
--webserver_interface=127.27.157.199
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:56.115403 2509 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:11:56.115772 2509 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:56.116659 2509 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:11:56.118217 1672 ts_manager.cc:194] Registered new tserver with Master: 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:11:56.119052 1672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.198:56437
I20240503 08:11:56.151456 2249 heartbeater.cc:503] Master 127.27.157.254:37795 was elected leader, sending a full tablet report...
W20240503 08:11:56.456934 2512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:56.457252 2512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:56.457376 2512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:56.465356 2512 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:56.465498 2512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:56.465665 2512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.199
I20240503 08:11:56.476120 2512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.157.199
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.199
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:56.477530 2512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:56.479437 2512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:56.492451 2517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.493031 2520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.493959 2518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:56.495038 2512 server_base.cc:1034] running on GCE node
I20240503 08:11:56.495872 2512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:56.496960 2512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:56.498186 2512 hybrid_clock.cc:648] HybridClock initialized: now 1714723916498131 us; error 59 us; skew 500 ppm
I20240503 08:11:56.498513 2512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:56.501070 2512 webserver.cc:460] Webserver started at http://127.27.157.199:41391/ using document root <none> and password file <none>
I20240503 08:11:56.501849 2512 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:56.501976 2512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:56.502296 2512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:56.505157 2512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "f24648c92c384f1ba349eee801f985e2"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:11:56.506006 2512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "f24648c92c384f1ba349eee801f985e2"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:11:56.511608 2512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240503 08:11:56.515420 2526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:56.516929 2512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 08:11:56.517118 2512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f24648c92c384f1ba349eee801f985e2"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:11:56.517336 2512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:56.533098 2512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:56.534005 2512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:56.534294 2512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:56.535370 2512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:56.537005 2512 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:56.537112 2512 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:56.537236 2512 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:56.537319 2512 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:56.571178 2512 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.199:39455
I20240503 08:11:56.571247 2638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.199:39455 every 8 connection(s)
I20240503 08:11:56.572739 2512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:11:56.582109 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 2512
I20240503 08:11:56.582324 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:11:56.585956 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.200:0
--local_ip_for_outbound_sockets=127.27.157.200
--webserver_interface=127.27.157.200
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:11:56.586548 2639 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:11:56.586864 2639 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:56.587785 2639 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:11:56.589229 1672 ts_manager.cc:194] Registered new tserver with Master: f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:11:56.590027 1672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.199:57299
I20240503 08:11:56.631021 2379 heartbeater.cc:503] Master 127.27.157.254:37795 was elected leader, sending a full tablet report...
W20240503 08:11:56.938225 2642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:11:56.938573 2642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:11:56.938716 2642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:11:56.946703 2642 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:11:56.946873 2642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:11:56.947033 2642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.200
I20240503 08:11:56.956529 2642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.157.200
--webserver_port=0
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.200
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:11:56.957940 2642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:11:56.959805 2642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:56.973096 2648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.973590 2650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:56.974323 2647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:56.975301 2642 server_base.cc:1034] running on GCE node
I20240503 08:11:56.976027 2642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:11:56.977106 2642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:11:56.978339 2642 hybrid_clock.cc:648] HybridClock initialized: now 1714723916978287 us; error 48 us; skew 500 ppm
I20240503 08:11:56.978673 2642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:11:56.981200 2642 webserver.cc:460] Webserver started at http://127.27.157.200:34781/ using document root <none> and password file <none>
I20240503 08:11:56.981983 2642 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:56.982111 2642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:56.982434 2642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:56.985342 2642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "00b61625d8a7460892c118a2570afda5"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:11:56.986168 2642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "00b61625d8a7460892c118a2570afda5"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:11:56.992314 2642 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20240503 08:11:56.996223 2656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:56.997785 2642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240503 08:11:56.997988 2642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "00b61625d8a7460892c118a2570afda5"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:11:56.998220 2642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:57.018810 2642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:57.019806 2642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:11:57.020094 2642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:57.021232 2642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:57.022857 2642 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:57.022969 2642 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:57.023087 2642 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:57.023159 2642 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:57.057502 2642 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.200:45283
I20240503 08:11:57.057560 2768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.200:45283 every 8 connection(s)
I20240503 08:11:57.058957 2642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:11:57.068606 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 2642
I20240503 08:11:57.068826 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:11:57.073210 2769 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:11:57.073539 2769 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:57.074470 2769 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:11:57.076009 1672 ts_manager.cc:194] Registered new tserver with Master: 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:11:57.076850 1672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.200:54415
I20240503 08:11:57.086007 28279 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:11:57.104954 28279 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:11:57.105129 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 2252
I20240503 08:11:57.113914 2374 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:57.122469 2509 heartbeater.cc:503] Master 127.27.157.254:37795 was elected leader, sending a full tablet report...
I20240503 08:11:57.199117 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 2252
I20240503 08:11:57.216289 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 2382
I20240503 08:11:57.224946 2504 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:57.308251 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 2382
I20240503 08:11:57.324785 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 2512
I20240503 08:11:57.333112 2634 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:57.415895 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 2512
I20240503 08:11:57.431893 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 2642
I20240503 08:11:57.440742 2764 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:11:57.523877 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 2642
I20240503 08:12:01.292059 1672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:01.333592 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 47800c04e5a74c548ce68cb42f74d78f (DEFAULT_TABLE table=rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:01.333962 1793 tablet_service.cc:1467] Processing CreateTablet for tablet a336330f849f4aca8d128a2647969f4e (DEFAULT_TABLE table=rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:01.334012 2184 tablet_service.cc:1467] Processing CreateTablet for tablet a336330f849f4aca8d128a2647969f4e (DEFAULT_TABLE table=rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:01.335456 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47800c04e5a74c548ce68cb42f74d78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:01.335439 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a336330f849f4aca8d128a2647969f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:01.335708 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 47800c04e5a74c548ce68cb42f74d78f (DEFAULT_TABLE table=rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:01.336604 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47800c04e5a74c548ce68cb42f74d78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:01.337954 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 31e125a3957b4a6298e5b2d7cae30bb9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:01.338889 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e125a3957b4a6298e5b2d7cae30bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:01.340029 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a336330f849f4aca8d128a2647969f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:01.354665 2798 tablet_bootstrap.cc:492] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:01.358916 2798 tablet_bootstrap.cc:654] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:01.359352 2054 tablet_service.cc:1467] Processing CreateTablet for tablet a336330f849f4aca8d128a2647969f4e (DEFAULT_TABLE table=rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:01.360404 2798 log.cc:824] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:01.361043 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a336330f849f4aca8d128a2647969f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:01.360582 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 31e125a3957b4a6298e5b2d7cae30bb9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:01.362457 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e125a3957b4a6298e5b2d7cae30bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:01.363534 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 47800c04e5a74c548ce68cb42f74d78f (DEFAULT_TABLE table=rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:01.363862 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 31e125a3957b4a6298e5b2d7cae30bb9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:01.364058 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47800c04e5a74c548ce68cb42f74d78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:01.364462 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e125a3957b4a6298e5b2d7cae30bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:01.374292 2800 tablet_bootstrap.cc:492] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:01.380159 2800 tablet_bootstrap.cc:654] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:01.381714 2800 log.cc:824] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:01.386118 2802 tablet_bootstrap.cc:492] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:01.394057 2803 tablet_bootstrap.cc:492] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:01.394286 2798 tablet_bootstrap.cc:492] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:01.394840 2802 tablet_bootstrap.cc:654] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:01.394941 2798 ts_tablet_manager.cc:1367] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.041s user 0.017s sys 0.019s
I20240503 08:12:01.396368 2802 log.cc:824] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:01.396821 2800 tablet_bootstrap.cc:492] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:01.397339 2800 ts_tablet_manager.cc:1367] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.003s
I20240503 08:12:01.398500 2803 tablet_bootstrap.cc:654] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:01.400153 2803 log.cc:824] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:01.406626 2802 tablet_bootstrap.cc:492] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:01.407120 2802 ts_tablet_manager.cc:1367] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.006s
I20240503 08:12:01.407475 2798 raft_consensus.cc:348] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:01.408329 2798 raft_consensus.cc:374] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:01.408546 2798 raft_consensus.cc:729] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:01.409751 2798 consensus_queue.cc:260] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:01.410718 2800 raft_consensus.cc:348] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:01.411365 2800 raft_consensus.cc:374] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:01.411543 2800 raft_consensus.cc:729] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:01.412523 2800 consensus_queue.cc:260] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:01.413661 2803 tablet_bootstrap.cc:492] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:01.414023 2803 ts_tablet_manager.cc:1367] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.000s
I20240503 08:12:01.418412 2798 ts_tablet_manager.cc:1398] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.023s user 0.012s sys 0.003s
I20240503 08:12:01.419453 2802 raft_consensus.cc:348] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:01.420306 2802 raft_consensus.cc:374] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:01.420521 2802 raft_consensus.cc:729] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:01.421777 2802 consensus_queue.cc:260] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:01.422102 2798 tablet_bootstrap.cc:492] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:01.424093 2798 tablet_bootstrap.cc:654] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:01.423949 2803 raft_consensus.cc:348] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:01.424571 2803 raft_consensus.cc:374] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:01.424751 2803 raft_consensus.cc:729] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:01.426333 2803 consensus_queue.cc:260] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:01.428205 2800 ts_tablet_manager.cc:1398] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.031s user 0.017s sys 0.008s
I20240503 08:12:01.430116 2802 ts_tablet_manager.cc:1398] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.023s user 0.014s sys 0.003s
I20240503 08:12:01.430665 2802 tablet_bootstrap.cc:492] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:01.432600 2802 tablet_bootstrap.cc:654] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:01.434876 2800 tablet_bootstrap.cc:492] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:01.436707 2798 tablet_bootstrap.cc:492] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:01.436967 2798 ts_tablet_manager.cc:1367] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240503 08:12:01.437510 2800 tablet_bootstrap.cc:654] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:01.438195 2798 raft_consensus.cc:348] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:01.438570 2798 raft_consensus.cc:374] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:01.438776 2798 raft_consensus.cc:729] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:01.439177 2798 consensus_queue.cc:260] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:01.440127 2798 ts_tablet_manager.cc:1398] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:01.442279 2803 ts_tablet_manager.cc:1398] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.028s user 0.019s sys 0.009s
I20240503 08:12:01.442921 2802 tablet_bootstrap.cc:492] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:01.443095 2802 ts_tablet_manager.cc:1367] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240503 08:12:01.443734 2803 tablet_bootstrap.cc:492] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:01.444463 2802 raft_consensus.cc:348] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:01.444815 2802 raft_consensus.cc:374] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:01.444998 2802 raft_consensus.cc:729] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:01.445448 2802 consensus_queue.cc:260] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:01.446144 2803 tablet_bootstrap.cc:654] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:01.447381 2802 ts_tablet_manager.cc:1398] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:01.449015 2803 tablet_bootstrap.cc:492] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:01.449194 2803 ts_tablet_manager.cc:1367] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 08:12:01.449726 2800 tablet_bootstrap.cc:492] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:01.449965 2800 ts_tablet_manager.cc:1367] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20240503 08:12:01.450644 2803 raft_consensus.cc:348] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:01.450886 2803 raft_consensus.cc:374] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:01.450997 2803 raft_consensus.cc:729] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:01.451244 2803 consensus_queue.cc:260] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:01.452229 2803 ts_tablet_manager.cc:1398] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:01.451048 2800 raft_consensus.cc:348] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:01.452500 2800 raft_consensus.cc:374] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:01.452610 2800 raft_consensus.cc:729] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:01.452873 2800 consensus_queue.cc:260] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:01.453817 2800 ts_tablet_manager.cc:1398] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:12:01.454186 2800 tablet_bootstrap.cc:492] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:01.455902 2800 tablet_bootstrap.cc:654] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:01.458225 2800 tablet_bootstrap.cc:492] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:01.458385 2800 ts_tablet_manager.cc:1367] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:01.458979 2800 raft_consensus.cc:348] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:01.459184 2800 raft_consensus.cc:374] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:01.459296 2800 raft_consensus.cc:729] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:01.459581 2800 consensus_queue.cc:260] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:01.460132 2800 ts_tablet_manager.cc:1398] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 08:12:01.474421 1859 tablet.cc:2369] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:01.486418 2807 raft_consensus.cc:482] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:01.486716 2807 raft_consensus.cc:504] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:01.489089 2807 leader_election.cc:288] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:01.497069 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47800c04e5a74c548ce68cb42f74d78f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:01.497071 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47800c04e5a74c548ce68cb42f74d78f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:01.497587 2204 raft_consensus.cc:2454] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:01.497596 1944 raft_consensus.cc:2454] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:01.498247 1747 leader_election.cc:302] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:01.498693 2807 raft_consensus.cc:2781] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:01.498868 2807 raft_consensus.cc:482] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:01.498988 2807 raft_consensus.cc:3037] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:01.501309 2807 raft_consensus.cc:504] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:01.501982 2807 leader_election.cc:288] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:01.502648 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47800c04e5a74c548ce68cb42f74d78f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:01.502852 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47800c04e5a74c548ce68cb42f74d78f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:01.502934 2204 raft_consensus.cc:3037] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:01.503074 1944 raft_consensus.cc:3037] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:01.506299 2204 raft_consensus.cc:2454] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:01.506307 1944 raft_consensus.cc:2454] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:01.506888 1747 leader_election.cc:302] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:01.507274 2807 raft_consensus.cc:2781] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:01.507759 2807 raft_consensus.cc:686] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:01.508299 2807 consensus_queue.cc:237] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:01.513415 1672 catalog_manager.cc:5484] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:01.529767 2808 raft_consensus.cc:482] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:01.530050 2808 raft_consensus.cc:504] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:01.532039 2808 leader_election.cc:288] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:01.539443 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a336330f849f4aca8d128a2647969f4e" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:01.539774 2204 raft_consensus.cc:2454] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:01.539633 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a336330f849f4aca8d128a2647969f4e" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:01.540144 1813 raft_consensus.cc:2454] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:01.540345 2008 leader_election.cc:302] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:01.540789 2808 raft_consensus.cc:2781] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:01.540944 2808 raft_consensus.cc:482] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:01.541051 2808 raft_consensus.cc:3037] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:01.543102 2808 raft_consensus.cc:504] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:01.543817 2808 leader_election.cc:288] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:01.544312 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a336330f849f4aca8d128a2647969f4e" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:01.544384 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a336330f849f4aca8d128a2647969f4e" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:01.544538 2204 raft_consensus.cc:3037] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:01.544587 1813 raft_consensus.cc:3037] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:01.546480 2204 raft_consensus.cc:2454] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:01.546636 1813 raft_consensus.cc:2454] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:01.547029 2008 leader_election.cc:302] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:01.547384 2808 raft_consensus.cc:2781] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:01.547825 2808 raft_consensus.cc:686] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:01.548353 2808 consensus_queue.cc:237] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:01.553072 1672 catalog_manager.cc:5484] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:01.567838 2807 raft_consensus.cc:482] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:01.568084 2807 raft_consensus.cc:504] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:01.568806 2807 leader_election.cc:288] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:01.569532 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e125a3957b4a6298e5b2d7cae30bb9" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:01.569870 1944 raft_consensus.cc:2454] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:01.570382 1748 leader_election.cc:302] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:01.570706 2807 raft_consensus.cc:2781] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:01.570792 2807 raft_consensus.cc:482] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:01.570891 2807 raft_consensus.cc:3037] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:01.572993 2807 raft_consensus.cc:504] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:01.573743 2807 leader_election.cc:288] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:01.574242 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e125a3957b4a6298e5b2d7cae30bb9" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:01.574455 1944 raft_consensus.cc:3037] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:01.576020 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e125a3957b4a6298e5b2d7cae30bb9" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:01.576017 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e125a3957b4a6298e5b2d7cae30bb9" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:01.576562 2073 raft_consensus.cc:3037] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:01.577332 1944 raft_consensus.cc:2454] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:01.577880 1748 leader_election.cc:302] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:01.578227 2807 raft_consensus.cc:2781] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:01.578408 2807 raft_consensus.cc:686] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:01.578662 2807 consensus_queue.cc:237] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:01.579409 2073 raft_consensus.cc:2454] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:01.581597 1672 catalog_manager.cc:5484] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:01.638789 2250 tablet.cc:2369] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:12:01.653729 2120 tablet.cc:2369] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:12:01.690554 1990 tablet.cc:2369] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:12:01.979568 2824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:01.979974 2824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:01.989109 2824 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:01.989125 2807 consensus_queue.cc:1035] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:01.998263 2807 consensus_queue.cc:1035] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:02.008209 2808 consensus_queue.cc:1035] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:02.015663 2842 consensus_queue.cc:1035] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 08:12:02.077932 2807 consensus_queue.cc:1035] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:02.085201 2859 consensus_queue.cc:1035] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 260.63 ms
time per row: 0.130315 ms
I20240503 08:12:02.423919 1672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:02.433959 2183 tablet_service.cc:1467] Processing CreateTablet for tablet b00417d7369a46aabfa933109c645c2c (DEFAULT_TABLE table=rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:02.434021 2053 tablet_service.cc:1467] Processing CreateTablet for tablet b00417d7369a46aabfa933109c645c2c (DEFAULT_TABLE table=rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:02.434389 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 314d235cdca241f8b80a7046fb1f524c (DEFAULT_TABLE table=rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:02.434510 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00417d7369a46aabfa933109c645c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.434526 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00417d7369a46aabfa933109c645c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.434803 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314d235cdca241f8b80a7046fb1f524c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.435443 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 314d235cdca241f8b80a7046fb1f524c (DEFAULT_TABLE table=rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:02.435874 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314d235cdca241f8b80a7046fb1f524c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.440435 1924 tablet_service.cc:1467] Processing CreateTablet for tablet b00417d7369a46aabfa933109c645c2c (DEFAULT_TABLE table=rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:02.440546 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 3219159f92774adbb7ecaa77ac267021 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:02.440948 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00417d7369a46aabfa933109c645c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.442026 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 314d235cdca241f8b80a7046fb1f524c (DEFAULT_TABLE table=rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:02.442191 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3219159f92774adbb7ecaa77ac267021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.442494 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314d235cdca241f8b80a7046fb1f524c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.445264 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 3219159f92774adbb7ecaa77ac267021 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:02.445537 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 3219159f92774adbb7ecaa77ac267021 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:02.445765 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3219159f92774adbb7ecaa77ac267021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.446009 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3219159f92774adbb7ecaa77ac267021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:02.459703 2888 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:02.461314 2889 tablet_bootstrap.cc:492] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:02.462404 2888 tablet_bootstrap.cc:654] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:02.463277 2887 tablet_bootstrap.cc:492] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:02.464038 2890 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:02.464563 2889 tablet_bootstrap.cc:654] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:02.465894 2888 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:02.466110 2888 ts_tablet_manager.cc:1367] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 08:12:02.466485 2890 tablet_bootstrap.cc:654] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:02.466509 2887 tablet_bootstrap.cc:654] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:02.467092 2888 raft_consensus.cc:348] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:02.467406 2888 raft_consensus.cc:374] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:02.467555 2888 raft_consensus.cc:729] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:02.467947 2888 consensus_queue.cc:260] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:02.469703 2888 ts_tablet_manager.cc:1398] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:02.470206 2888 tablet_bootstrap.cc:492] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:02.472843 2888 tablet_bootstrap.cc:654] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:02.473454 2889 tablet_bootstrap.cc:492] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:02.473697 2889 ts_tablet_manager.cc:1367] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20240503 08:12:02.474722 2889 raft_consensus.cc:348] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:02.475071 2889 raft_consensus.cc:374] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:02.475224 2889 raft_consensus.cc:729] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:02.475661 2889 consensus_queue.cc:260] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:02.476544 2889 ts_tablet_manager.cc:1398] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:02.477805 2889 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:02.479846 2889 tablet_bootstrap.cc:654] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:02.481389 2887 tablet_bootstrap.cc:492] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:02.482125 2887 ts_tablet_manager.cc:1367] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20240503 08:12:02.483373 2887 raft_consensus.cc:348] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:02.483681 2887 raft_consensus.cc:374] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:02.483776 2887 raft_consensus.cc:729] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:02.484151 2887 consensus_queue.cc:260] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:02.484468 2890 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:02.485411 2890 ts_tablet_manager.cc:1367] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s
I20240503 08:12:02.486351 2890 raft_consensus.cc:348] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:02.486573 2890 raft_consensus.cc:374] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:02.486662 2890 raft_consensus.cc:729] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:02.486804 2887 ts_tablet_manager.cc:1398] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:02.486994 2890 consensus_queue.cc:260] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:02.487275 2887 tablet_bootstrap.cc:492] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:02.487874 2890 ts_tablet_manager.cc:1398] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:02.490022 2888 tablet_bootstrap.cc:492] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:02.490255 2888 ts_tablet_manager.cc:1367] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.011s
I20240503 08:12:02.491231 2887 tablet_bootstrap.cc:654] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:02.491261 2889 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:02.491168 2888 raft_consensus.cc:348] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:02.491430 2888 raft_consensus.cc:374] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:02.491510 2889 ts_tablet_manager.cc:1367] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20240503 08:12:02.491526 2888 raft_consensus.cc:729] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:02.491865 2888 consensus_queue.cc:260] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:02.492585 2888 ts_tablet_manager.cc:1398] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:02.492931 2888 tablet_bootstrap.cc:492] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:02.493878 2889 raft_consensus.cc:348] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:02.494652 2888 tablet_bootstrap.cc:654] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:02.495167 2889 raft_consensus.cc:374] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:02.495345 2889 raft_consensus.cc:729] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:02.497985 2889 consensus_queue.cc:260] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:02.500918 2888 tablet_bootstrap.cc:492] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:02.501058 2889 ts_tablet_manager.cc:1398] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20240503 08:12:02.501142 2888 ts_tablet_manager.cc:1367] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:12:02.501607 2889 tablet_bootstrap.cc:492] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:02.502063 2888 raft_consensus.cc:348] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:02.502301 2888 raft_consensus.cc:374] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:02.502410 2888 raft_consensus.cc:729] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:02.502704 2888 consensus_queue.cc:260] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:02.503481 2888 ts_tablet_manager.cc:1398] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:02.504173 2889 tablet_bootstrap.cc:654] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:02.505053 2887 tablet_bootstrap.cc:492] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:02.505383 2887 ts_tablet_manager.cc:1367] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240503 08:12:02.506114 2887 raft_consensus.cc:348] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:02.506330 2887 raft_consensus.cc:374] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:02.506412 2887 raft_consensus.cc:729] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:02.506753 2887 consensus_queue.cc:260] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:02.506915 2889 tablet_bootstrap.cc:492] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:02.507071 2889 ts_tablet_manager.cc:1367] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:12:02.507990 2887 ts_tablet_manager.cc:1398] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:02.508000 2889 raft_consensus.cc:348] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:02.508450 2889 raft_consensus.cc:374] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:02.508625 2889 raft_consensus.cc:729] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:02.508965 2889 consensus_queue.cc:260] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:02.509665 2889 ts_tablet_manager.cc:1398] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:02.543149 2842 raft_consensus.cc:482] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:02.543464 2842 raft_consensus.cc:504] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:02.544409 2842 leader_election.cc:288] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:02.545025 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00417d7369a46aabfa933109c645c2c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:02.545323 2204 raft_consensus.cc:2454] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:02.545913 2008 leader_election.cc:302] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:02.546283 2842 raft_consensus.cc:2781] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:02.546375 2842 raft_consensus.cc:482] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:02.546494 2842 raft_consensus.cc:3037] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:02.549211 2842 raft_consensus.cc:504] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:02.550089 2842 leader_election.cc:288] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:02.550663 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00417d7369a46aabfa933109c645c2c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:02.550908 2204 raft_consensus.cc:3037] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:02.553149 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00417d7369a46aabfa933109c645c2c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:02.553133 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00417d7369a46aabfa933109c645c2c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:02.553331 2204 raft_consensus.cc:2454] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:02.553458 1943 raft_consensus.cc:2454] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:02.553856 2008 leader_election.cc:302] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:02.554174 2842 raft_consensus.cc:2781] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:02.554371 2842 raft_consensus.cc:686] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:02.554662 2842 consensus_queue.cc:237] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:02.557628 1668 catalog_manager.cc:5484] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:02.595602 2859 raft_consensus.cc:482] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:02.595887 2859 raft_consensus.cc:504] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:02.596757 2859 leader_election.cc:288] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:02.597426 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3219159f92774adbb7ecaa77ac267021" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:02.597395 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3219159f92774adbb7ecaa77ac267021" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:02.597702 1944 raft_consensus.cc:2454] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:02.597702 2073 raft_consensus.cc:2454] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:02.598219 1748 leader_election.cc:302] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:02.598548 2859 raft_consensus.cc:2781] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:02.598640 2859 raft_consensus.cc:482] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:02.598752 2859 raft_consensus.cc:3037] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:02.600780 2859 raft_consensus.cc:504] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:02.601446 2859 leader_election.cc:288] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:02.602083 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3219159f92774adbb7ecaa77ac267021" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:02.602083 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3219159f92774adbb7ecaa77ac267021" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:02.602348 2073 raft_consensus.cc:3037] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:02.602365 1944 raft_consensus.cc:3037] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:02.605387 2073 raft_consensus.cc:2454] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:02.605387 1944 raft_consensus.cc:2454] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:02.605971 1747 leader_election.cc:302] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:02.606339 2859 raft_consensus.cc:2781] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:02.606504 2859 raft_consensus.cc:686] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:02.606767 2859 consensus_queue.cc:237] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:02.609853 1668 catalog_manager.cc:5484] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:02.626353 2832 raft_consensus.cc:482] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:02.626581 2832 raft_consensus.cc:504] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:02.628360 2832 leader_election.cc:288] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:02.635918 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314d235cdca241f8b80a7046fb1f524c" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:02.635897 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314d235cdca241f8b80a7046fb1f524c" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:02.636152 2073 raft_consensus.cc:2454] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:02.636144 1944 raft_consensus.cc:2454] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:02.636684 2138 leader_election.cc:302] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:02.637120 2832 raft_consensus.cc:2781] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:02.637267 2832 raft_consensus.cc:482] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:02.637372 2832 raft_consensus.cc:3037] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:02.639461 2832 raft_consensus.cc:504] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:02.640157 2832 leader_election.cc:288] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:02.640715 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314d235cdca241f8b80a7046fb1f524c" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:02.640777 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314d235cdca241f8b80a7046fb1f524c" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:02.640982 1944 raft_consensus.cc:3037] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:02.640985 2073 raft_consensus.cc:3037] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:02.644119 2073 raft_consensus.cc:2454] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:02.644119 1944 raft_consensus.cc:2454] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:02.644765 2139 leader_election.cc:302] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:02.645195 2832 raft_consensus.cc:2781] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:02.645678 2832 raft_consensus.cc:686] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:02.646215 2832 consensus_queue.cc:237] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:02.650913 1668 catalog_manager.cc:5484] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:02.960039 2842 consensus_queue.cc:1035] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:02.965320 1944 raft_consensus.cc:3037] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:02.969488 2808 consensus_queue.cc:1035] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:03.054215 2908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:03.054630 2908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:03.062541 2908 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:12:03.161104 2859 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:03.165278 2832 consensus_queue.cc:1035] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:03.193255 2936 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:03.223747 2832 consensus_queue.cc:1035] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 305.665 ms
time per row: 0.152833 ms
I20240503 08:12:03.522104 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:03.531657 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 6f932635e2bf4c9da5f5f194975274a3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:03.532079 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 6f932635e2bf4c9da5f5f194975274a3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:03.532217 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f932635e2bf4c9da5f5f194975274a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.532519 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f932635e2bf4c9da5f5f194975274a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.532578 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 6f932635e2bf4c9da5f5f194975274a3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:03.533072 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 7071eecd95ca42159859ff358fcf9119 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:03.533087 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f932635e2bf4c9da5f5f194975274a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.534652 2182 tablet_service.cc:1467] Processing CreateTablet for tablet a82d95d9203d41348c46fc95a4cd75fd (DEFAULT_TABLE table=rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:03.535141 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82d95d9203d41348c46fc95a4cd75fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.538288 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7071eecd95ca42159859ff358fcf9119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.539641 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 7071eecd95ca42159859ff358fcf9119 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:03.540236 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7071eecd95ca42159859ff358fcf9119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.543861 1791 tablet_service.cc:1467] Processing CreateTablet for tablet a82d95d9203d41348c46fc95a4cd75fd (DEFAULT_TABLE table=rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:03.544449 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82d95d9203d41348c46fc95a4cd75fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.544535 2972 tablet_bootstrap.cc:492] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:03.546486 2972 tablet_bootstrap.cc:654] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:03.546725 2973 tablet_bootstrap.cc:492] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:03.547737 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 7071eecd95ca42159859ff358fcf9119 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:03.548154 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7071eecd95ca42159859ff358fcf9119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.548707 2973 tablet_bootstrap.cc:654] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:03.551985 1923 tablet_service.cc:1467] Processing CreateTablet for tablet a82d95d9203d41348c46fc95a4cd75fd (DEFAULT_TABLE table=rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:03.552661 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82d95d9203d41348c46fc95a4cd75fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:03.554221 2973 tablet_bootstrap.cc:492] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:03.554316 2976 tablet_bootstrap.cc:492] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:03.554396 2973 ts_tablet_manager.cc:1367] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:12:03.555377 2973 raft_consensus.cc:348] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:03.555670 2973 raft_consensus.cc:374] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:03.555799 2973 raft_consensus.cc:729] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:03.556197 2973 consensus_queue.cc:260] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:03.556924 2973 ts_tablet_manager.cc:1398] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:03.557832 2976 tablet_bootstrap.cc:654] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:03.580091 2973 tablet_bootstrap.cc:492] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:03.581876 2978 tablet_bootstrap.cc:492] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:03.582653 2973 tablet_bootstrap.cc:654] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:03.585665 2978 tablet_bootstrap.cc:654] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:03.586089 2973 tablet_bootstrap.cc:492] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:03.586292 2973 ts_tablet_manager.cc:1367] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240503 08:12:03.587364 2972 tablet_bootstrap.cc:492] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:03.587217 2973 raft_consensus.cc:348] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:03.587500 2973 raft_consensus.cc:374] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:03.587582 2972 ts_tablet_manager.cc:1367] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.043s user 0.019s sys 0.020s
I20240503 08:12:03.587582 2973 raft_consensus.cc:729] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:03.587847 2973 consensus_queue.cc:260] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:03.588440 2973 ts_tablet_manager.cc:1398] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:03.589032 2976 tablet_bootstrap.cc:492] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:03.588405 2972 raft_consensus.cc:348] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:03.589287 2976 ts_tablet_manager.cc:1367] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.029s
I20240503 08:12:03.589291 2972 raft_consensus.cc:374] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:03.589390 2972 raft_consensus.cc:729] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:03.589728 2972 consensus_queue.cc:260] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:03.590202 2976 raft_consensus.cc:348] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:03.590358 2972 ts_tablet_manager.cc:1398] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:03.590423 2978 tablet_bootstrap.cc:492] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:03.590490 2976 raft_consensus.cc:374] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:03.590611 2978 ts_tablet_manager.cc:1367] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 08:12:03.590615 2976 raft_consensus.cc:729] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:03.590896 2976 consensus_queue.cc:260] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:03.591432 2972 tablet_bootstrap.cc:492] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:03.591583 2976 ts_tablet_manager.cc:1398] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:03.591576 2978 raft_consensus.cc:348] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:03.591821 2978 raft_consensus.cc:374] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:03.591943 2978 raft_consensus.cc:729] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:03.592242 2978 consensus_queue.cc:260] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:03.593860 2972 tablet_bootstrap.cc:654] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:03.594835 2978 ts_tablet_manager.cc:1398] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:03.595307 2978 tablet_bootstrap.cc:492] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:03.597497 2978 tablet_bootstrap.cc:654] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:03.599821 2972 tablet_bootstrap.cc:492] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:03.599908 2978 tablet_bootstrap.cc:492] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:03.600044 2972 ts_tablet_manager.cc:1367] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 08:12:03.600080 2978 ts_tablet_manager.cc:1367] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 08:12:03.600728 2978 raft_consensus.cc:348] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:03.600728 2972 raft_consensus.cc:348] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:03.600912 2978 raft_consensus.cc:374] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:03.600947 2972 raft_consensus.cc:374] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:03.601020 2978 raft_consensus.cc:729] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:03.601034 2972 raft_consensus.cc:729] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:03.601325 2978 consensus_queue.cc:260] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:03.601296 2972 consensus_queue.cc:260] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:03.601837 2978 ts_tablet_manager.cc:1398] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:03.601938 2972 ts_tablet_manager.cc:1398] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:03.602141 2978 tablet_bootstrap.cc:492] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:03.602420 2972 tablet_bootstrap.cc:492] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:03.603976 2978 tablet_bootstrap.cc:654] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:03.604121 2972 tablet_bootstrap.cc:654] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:03.605962 2972 tablet_bootstrap.cc:492] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:03.606173 2972 ts_tablet_manager.cc:1367] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:03.606818 2978 tablet_bootstrap.cc:492] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:03.606971 2978 ts_tablet_manager.cc:1367] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:12:03.607014 2972 raft_consensus.cc:348] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:03.607275 2972 raft_consensus.cc:374] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:03.607451 2972 raft_consensus.cc:729] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:03.607709 2972 consensus_queue.cc:260] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:03.607776 2978 raft_consensus.cc:348] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:03.608014 2978 raft_consensus.cc:374] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:03.608131 2978 raft_consensus.cc:729] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:03.608256 2972 ts_tablet_manager.cc:1398] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:03.608368 2978 consensus_queue.cc:260] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:03.608939 2978 ts_tablet_manager.cc:1398] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:03.646800 2859 raft_consensus.cc:482] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:03.647033 2859 raft_consensus.cc:504] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:03.647814 2859 leader_election.cc:288] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:03.648463 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7071eecd95ca42159859ff358fcf9119" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:03.648460 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7071eecd95ca42159859ff358fcf9119" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:03.648730 2204 raft_consensus.cc:2454] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:03.648739 2073 raft_consensus.cc:2454] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:03.649197 1747 leader_election.cc:302] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:03.649498 2859 raft_consensus.cc:2781] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:03.649605 2859 raft_consensus.cc:482] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.649724 2859 raft_consensus.cc:3037] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:03.651755 2859 raft_consensus.cc:504] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:03.652467 2859 leader_election.cc:288] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:03.652984 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7071eecd95ca42159859ff358fcf9119" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:03.653223 2204 raft_consensus.cc:3037] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:03.653182 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7071eecd95ca42159859ff358fcf9119" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:03.653409 2073 raft_consensus.cc:3037] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:03.655181 2204 raft_consensus.cc:2454] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:03.655735 1747 leader_election.cc:302] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:03.656044 2859 raft_consensus.cc:2781] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:03.656271 2859 raft_consensus.cc:686] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:03.656463 2073 raft_consensus.cc:2454] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:03.656569 2859 consensus_queue.cc:237] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:03.659726 1666 catalog_manager.cc:5484] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:03.752177 2859 raft_consensus.cc:482] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:03.752372 2859 raft_consensus.cc:504] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:03.753098 2859 leader_election.cc:288] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:03.753661 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82d95d9203d41348c46fc95a4cd75fd" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:03.753669 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82d95d9203d41348c46fc95a4cd75fd" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:03.753901 2204 raft_consensus.cc:2454] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:03.753908 1943 raft_consensus.cc:2454] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:03.754326 1747 leader_election.cc:302] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:03.754631 2859 raft_consensus.cc:2781] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:03.754763 2859 raft_consensus.cc:482] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.754877 2859 raft_consensus.cc:3037] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:03.754897 2828 raft_consensus.cc:482] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:03.755070 2828 raft_consensus.cc:504] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:03.757055 2828 leader_election.cc:288] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:03.758478 2859 raft_consensus.cc:504] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:03.759792 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82d95d9203d41348c46fc95a4cd75fd" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:03.760046 2204 raft_consensus.cc:3037] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:03.760061 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82d95d9203d41348c46fc95a4cd75fd" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:03.760290 1943 raft_consensus.cc:3037] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:03.760455 2859 leader_election.cc:288] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:03.763523 1943 raft_consensus.cc:2454] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:03.764163 1748 leader_election.cc:302] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:03.764499 2204 raft_consensus.cc:2454] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:03.765527 2859 raft_consensus.cc:2781] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:03.765904 2859 raft_consensus.cc:686] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:03.766566 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f932635e2bf4c9da5f5f194975274a3" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:03.766674 2859 consensus_queue.cc:237] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:03.766803 1812 raft_consensus.cc:2454] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:03.766979 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f932635e2bf4c9da5f5f194975274a3" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:03.767233 2204 raft_consensus.cc:2454] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:03.767416 1879 leader_election.cc:302] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:03.767858 2828 raft_consensus.cc:2781] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:03.767984 2828 raft_consensus.cc:482] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.768082 2828 raft_consensus.cc:3037] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:03.770346 2828 raft_consensus.cc:504] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:03.771018 2828 leader_election.cc:288] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:03.771533 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f932635e2bf4c9da5f5f194975274a3" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:03.771744 1812 raft_consensus.cc:3037] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:03.772636 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f932635e2bf4c9da5f5f194975274a3" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:03.772894 2204 raft_consensus.cc:3037] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:03.774565 1812 raft_consensus.cc:2454] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:03.774761 2204 raft_consensus.cc:2454] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:03.774700 1666 catalog_manager.cc:5484] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:03.775068 1879 leader_election.cc:302] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:03.775449 2828 raft_consensus.cc:2781] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:03.775933 2828 raft_consensus.cc:686] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:03.776985 2828 consensus_queue.cc:237] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:03.780884 1668 catalog_manager.cc:5484] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:04.165848 2991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:04.166189 2991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:04.174002 2991 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:04.179260 2859 consensus_queue.cc:1035] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:04.196075 2941 consensus_queue.cc:1035] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 08:12:04.255653 2828 consensus_queue.cc:1035] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:04.256793 2941 consensus_queue.cc:1035] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:04.268915 2941 consensus_queue.cc:1035] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:04.303992 3025 consensus_queue.cc:1035] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 253.055 ms
time per row: 0.126528 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:05.552858 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:05.562408 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc8279b96f74e879b499b0143502c8f (DEFAULT_TABLE table=rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:05.562930 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc8279b96f74e879b499b0143502c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:05.562825 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc8279b96f74e879b499b0143502c8f (DEFAULT_TABLE table=rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:05.563246 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc8279b96f74e879b499b0143502c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:05.563680 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc8279b96f74e879b499b0143502c8f (DEFAULT_TABLE table=rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:05.563777 2052 tablet_service.cc:1467] Processing CreateTablet for tablet fede923140fb45a5a4f93c0846b62595 (DEFAULT_TABLE table=rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:05.564143 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc8279b96f74e879b499b0143502c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:05.564141 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fede923140fb45a5a4f93c0846b62595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:05.571645 1793 tablet_service.cc:1467] Processing CreateTablet for tablet fede923140fb45a5a4f93c0846b62595 (DEFAULT_TABLE table=rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:05.571646 2183 tablet_service.cc:1467] Processing CreateTablet for tablet fede923140fb45a5a4f93c0846b62595 (DEFAULT_TABLE table=rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:05.572120 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fede923140fb45a5a4f93c0846b62595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:05.572118 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fede923140fb45a5a4f93c0846b62595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:05.575831 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 266d0c75717b4fc98eae8b75a5ba4d8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:05.576890 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 266d0c75717b4fc98eae8b75a5ba4d8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:05.577404 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266d0c75717b4fc98eae8b75a5ba4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:05.578305 3064 tablet_bootstrap.cc:492] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:05.578953 3063 tablet_bootstrap.cc:492] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:05.579898 3064 tablet_bootstrap.cc:654] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:05.580690 3063 tablet_bootstrap.cc:654] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:05.582737 3064 tablet_bootstrap.cc:492] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:05.582911 3064 ts_tablet_manager.cc:1367] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:05.583657 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266d0c75717b4fc98eae8b75a5ba4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:05.583803 3064 raft_consensus.cc:348] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:05.584010 3064 raft_consensus.cc:374] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:05.584128 3064 raft_consensus.cc:729] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:05.584408 3064 consensus_queue.cc:260] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:05.585677 3064 ts_tablet_manager.cc:1398] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:05.585927 3064 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:05.587136 3064 tablet_bootstrap.cc:654] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:05.589608 2958 raft_consensus.cc:482] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:05.589773 2958 raft_consensus.cc:504] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:05.590545 2958 leader_election.cc:288] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:05.592451 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc8279b96f74e879b499b0143502c8f" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
W20240503 08:12:05.594082 2009 leader_election.cc:341] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:12:05.594386 2009 leader_election.cc:341] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Not found: Tablet not found: 8fc8279b96f74e879b499b0143502c8f
I20240503 08:12:05.594496 2009 leader_election.cc:302] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e
I20240503 08:12:05.595399 2958 raft_consensus.cc:2726] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:05.596930 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 266d0c75717b4fc98eae8b75a5ba4d8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:05.597766 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266d0c75717b4fc98eae8b75a5ba4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:05.601636 3064 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:05.601904 3064 ts_tablet_manager.cc:1367] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240503 08:12:05.605223 3064 raft_consensus.cc:348] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:05.605633 3064 raft_consensus.cc:374] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:05.605737 3064 raft_consensus.cc:729] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:05.606160 3064 consensus_queue.cc:260] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:05.600178 3068 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:05.610354 3064 ts_tablet_manager.cc:1398] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 08:12:05.610738 3064 tablet_bootstrap.cc:492] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:05.611111 3063 tablet_bootstrap.cc:492] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:05.611369 3063 ts_tablet_manager.cc:1367] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.024s
I20240503 08:12:05.612262 3063 raft_consensus.cc:348] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:05.612567 3064 tablet_bootstrap.cc:654] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:05.612599 3063 raft_consensus.cc:374] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:05.612805 3063 raft_consensus.cc:729] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:05.613130 3063 consensus_queue.cc:260] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:05.613972 3068 tablet_bootstrap.cc:654] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:05.614575 3063 ts_tablet_manager.cc:1398] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:05.614907 3063 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:05.617365 3069 tablet_bootstrap.cc:492] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:05.618152 3064 tablet_bootstrap.cc:492] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:05.618338 3064 ts_tablet_manager.cc:1367] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 08:12:05.618573 3063 tablet_bootstrap.cc:654] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:05.619143 3069 tablet_bootstrap.cc:654] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:05.619190 3064 raft_consensus.cc:348] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:05.619427 3064 raft_consensus.cc:374] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:05.619544 3064 raft_consensus.cc:729] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:05.619869 3064 consensus_queue.cc:260] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:05.620461 3064 ts_tablet_manager.cc:1398] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:05.622123 3068 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:05.622345 3068 ts_tablet_manager.cc:1367] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240503 08:12:05.623520 3068 raft_consensus.cc:348] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:05.623844 3068 raft_consensus.cc:374] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:05.623977 3068 raft_consensus.cc:729] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:05.624292 3068 consensus_queue.cc:260] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:05.625280 3068 ts_tablet_manager.cc:1398] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:12:05.625679 3068 tablet_bootstrap.cc:492] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:05.628106 3068 tablet_bootstrap.cc:654] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:05.630141 3069 tablet_bootstrap.cc:492] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:05.630332 3069 ts_tablet_manager.cc:1367] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 08:12:05.630502 3063 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:05.630647 3068 tablet_bootstrap.cc:492] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:05.630689 3063 ts_tablet_manager.cc:1367] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.005s
I20240503 08:12:05.630837 3068 ts_tablet_manager.cc:1367] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:12:05.631212 3069 raft_consensus.cc:348] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:05.631605 3069 raft_consensus.cc:374] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:05.631539 3063 raft_consensus.cc:348] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:05.631718 3069 raft_consensus.cc:729] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:05.631822 3063 raft_consensus.cc:374] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:05.631902 3063 raft_consensus.cc:729] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:05.631763 3068 raft_consensus.cc:348] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:05.632007 3068 raft_consensus.cc:374] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:05.632097 3068 raft_consensus.cc:729] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:05.631989 3069 consensus_queue.cc:260] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:05.632144 3063 consensus_queue.cc:260] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:05.632402 3068 consensus_queue.cc:260] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:05.632625 3069 ts_tablet_manager.cc:1398] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:05.632961 3069 tablet_bootstrap.cc:492] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:05.633482 3063 ts_tablet_manager.cc:1398] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:05.635679 3068 ts_tablet_manager.cc:1398] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:05.636005 3069 tablet_bootstrap.cc:654] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:05.639488 3069 tablet_bootstrap.cc:492] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:05.639707 3069 ts_tablet_manager.cc:1367] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 08:12:05.640343 3069 raft_consensus.cc:348] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:05.640537 3069 raft_consensus.cc:374] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:05.640585 3069 raft_consensus.cc:729] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:05.640798 3069 consensus_queue.cc:260] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:05.641258 3069 ts_tablet_manager.cc:1398] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:05.645464 3036 raft_consensus.cc:482] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:05.645663 3036 raft_consensus.cc:504] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:05.646335 3036 leader_election.cc:288] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:05.646917 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fede923140fb45a5a4f93c0846b62595" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:05.646929 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fede923140fb45a5a4f93c0846b62595" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:05.647119 2073 raft_consensus.cc:2454] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:05.647190 2204 raft_consensus.cc:2454] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:05.647624 2958 raft_consensus.cc:482] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:05.647701 1747 leader_election.cc:302] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:05.647789 2958 raft_consensus.cc:504] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:05.648006 3036 raft_consensus.cc:2781] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:05.648119 3036 raft_consensus.cc:482] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.648206 3036 raft_consensus.cc:3037] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:05.648626 2958 leader_election.cc:288] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:05.649027 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266d0c75717b4fc98eae8b75a5ba4d8c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:05.649255 1944 raft_consensus.cc:2454] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:05.650537 3036 raft_consensus.cc:504] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:05.651098 3036 leader_election.cc:288] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:05.651846 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fede923140fb45a5a4f93c0846b62595" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:05.651888 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266d0c75717b4fc98eae8b75a5ba4d8c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:05.652009 2073 raft_consensus.cc:3037] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:05.652143 2203 raft_consensus.cc:2454] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:05.652508 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fede923140fb45a5a4f93c0846b62595" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:05.652660 2204 raft_consensus.cc:3037] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:05.652693 2008 leader_election.cc:302] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:05.653018 2958 raft_consensus.cc:2781] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:05.653136 2958 raft_consensus.cc:482] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.653239 2958 raft_consensus.cc:3037] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:05.655256 2073 raft_consensus.cc:2454] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:05.655545 2204 raft_consensus.cc:2454] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:05.655826 1747 leader_election.cc:302] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:05.656234 2958 raft_consensus.cc:504] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:05.657171 2958 leader_election.cc:288] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:05.657171 3036 raft_consensus.cc:2781] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:05.657594 3036 raft_consensus.cc:686] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:05.657579 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266d0c75717b4fc98eae8b75a5ba4d8c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:05.657824 2204 raft_consensus.cc:3037] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:05.658222 3036 consensus_queue.cc:237] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:05.658396 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266d0c75717b4fc98eae8b75a5ba4d8c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:05.658605 1944 raft_consensus.cc:3037] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:05.660316 2204 raft_consensus.cc:2454] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:05.660831 2008 leader_election.cc:302] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:05.661381 2958 raft_consensus.cc:2781] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:05.661403 1944 raft_consensus.cc:2454] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:05.661904 2958 raft_consensus.cc:686] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:05.662658 2958 consensus_queue.cc:237] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:05.663034 1668 catalog_manager.cc:5484] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:05.666136 1666 catalog_manager.cc:5484] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:05.758132 3036 raft_consensus.cc:482] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:05.758359 3036 raft_consensus.cc:504] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:05.758973 3036 leader_election.cc:288] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:05.759644 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc8279b96f74e879b499b0143502c8f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:05.759649 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc8279b96f74e879b499b0143502c8f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:05.759917 1944 raft_consensus.cc:2454] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:05.759932 2073 raft_consensus.cc:2454] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:05.760391 1748 leader_election.cc:302] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:05.760679 3036 raft_consensus.cc:2781] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:05.760776 3036 raft_consensus.cc:482] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.760906 3036 raft_consensus.cc:3037] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:05.762876 3036 raft_consensus.cc:504] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:05.763587 3036 leader_election.cc:288] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:05.764125 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc8279b96f74e879b499b0143502c8f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:05.764359 1944 raft_consensus.cc:3037] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:05.764360 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc8279b96f74e879b499b0143502c8f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:05.764605 2073 raft_consensus.cc:3037] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:05.766429 1944 raft_consensus.cc:2454] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:05.766541 2073 raft_consensus.cc:2454] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:05.766923 1748 leader_election.cc:302] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:05.767292 3036 raft_consensus.cc:2781] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:05.767493 3036 raft_consensus.cc:686] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:05.767802 3036 consensus_queue.cc:237] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:05.770910 1666 catalog_manager.cc:5484] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:06.056020 2958 consensus_queue.cc:1035] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:06.065161 3084 consensus_queue.cc:1035] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:06.108217 3036 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:06.115968 3036 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:06.181564 3081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:06.181953 3081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:06.194298 3081 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 08:12:06.299571 3036 consensus_queue.cc:1035] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:06.305387 3090 consensus_queue.cc:1035] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 231.719 ms
time per row: 0.11586 ms
I20240503 08:12:06.582027 1663 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:06.591893 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 23e53aef15cc4b80b1793a6a10eb5b2b (DEFAULT_TABLE table=rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:06.591984 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 23e53aef15cc4b80b1793a6a10eb5b2b (DEFAULT_TABLE table=rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:06.592411 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23e53aef15cc4b80b1793a6a10eb5b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:06.592474 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23e53aef15cc4b80b1793a6a10eb5b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:06.592813 2054 tablet_service.cc:1467] Processing CreateTablet for tablet f5408b903cd14f3384a68a12fe4abcbb (DEFAULT_TABLE table=rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:06.593259 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5408b903cd14f3384a68a12fe4abcbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:06.593845 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 23e53aef15cc4b80b1793a6a10eb5b2b (DEFAULT_TABLE table=rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:06.594302 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23e53aef15cc4b80b1793a6a10eb5b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:06.599553 1924 tablet_service.cc:1467] Processing CreateTablet for tablet f5408b903cd14f3384a68a12fe4abcbb (DEFAULT_TABLE table=rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:06.600057 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5408b903cd14f3384a68a12fe4abcbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:06.600351 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 7890ae8e9d7d4b4ba072de7969c20fb0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:06.600347 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 7890ae8e9d7d4b4ba072de7969c20fb0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:06.600744 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7890ae8e9d7d4b4ba072de7969c20fb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:06.600751 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7890ae8e9d7d4b4ba072de7969c20fb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:06.604318 1793 tablet_service.cc:1467] Processing CreateTablet for tablet f5408b903cd14f3384a68a12fe4abcbb (DEFAULT_TABLE table=rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:06.604799 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5408b903cd14f3384a68a12fe4abcbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:06.607784 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 7890ae8e9d7d4b4ba072de7969c20fb0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:06.608253 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7890ae8e9d7d4b4ba072de7969c20fb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:06.614146 3145 tablet_bootstrap.cc:492] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:06.618206 3145 tablet_bootstrap.cc:654] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:06.621160 3146 tablet_bootstrap.cc:492] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:06.624042 3146 tablet_bootstrap.cc:654] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:06.627303 3146 tablet_bootstrap.cc:492] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:06.627579 3146 ts_tablet_manager.cc:1367] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 08:12:06.628715 3146 raft_consensus.cc:348] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:06.629026 3146 raft_consensus.cc:374] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:06.629143 3146 raft_consensus.cc:729] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:06.629272 3150 tablet_bootstrap.cc:492] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:06.629536 3146 consensus_queue.cc:260] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:06.630349 3145 tablet_bootstrap.cc:492] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:06.631218 3150 tablet_bootstrap.cc:654] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:06.632125 3145 ts_tablet_manager.cc:1367] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240503 08:12:06.633141 3145 raft_consensus.cc:348] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:06.633478 3145 raft_consensus.cc:374] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:06.633565 3147 tablet_bootstrap.cc:492] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:06.633679 3145 raft_consensus.cc:729] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:06.634057 3146 ts_tablet_manager.cc:1398] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:06.634025 3145 consensus_queue.cc:260] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:06.634410 3146 tablet_bootstrap.cc:492] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:06.635481 3147 tablet_bootstrap.cc:654] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:06.635469 3145 ts_tablet_manager.cc:1398] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:06.635763 3145 tablet_bootstrap.cc:492] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:06.636334 3146 tablet_bootstrap.cc:654] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:06.637502 3150 tablet_bootstrap.cc:492] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:06.637662 3150 ts_tablet_manager.cc:1367] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:12:06.638530 3150 raft_consensus.cc:348] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.638774 3150 raft_consensus.cc:374] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:06.638867 3150 raft_consensus.cc:729] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:06.639159 3150 consensus_queue.cc:260] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.639895 3150 ts_tablet_manager.cc:1398] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 08:12:06.640197 3150 tablet_bootstrap.cc:492] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:06.642069 3150 tablet_bootstrap.cc:654] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:06.642135 3145 tablet_bootstrap.cc:654] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:06.645258 3147 tablet_bootstrap.cc:492] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:06.645470 3147 ts_tablet_manager.cc:1367] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240503 08:12:06.646425 3147 raft_consensus.cc:348] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:06.646617 3147 raft_consensus.cc:374] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:06.646684 3145 tablet_bootstrap.cc:492] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:06.646718 3147 raft_consensus.cc:729] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:06.646854 3145 ts_tablet_manager.cc:1367] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 08:12:06.647015 3147 consensus_queue.cc:260] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:06.647702 3145 raft_consensus.cc:348] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.647922 3145 raft_consensus.cc:374] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:06.647992 3145 raft_consensus.cc:729] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:06.648278 3145 consensus_queue.cc:260] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.648913 3145 ts_tablet_manager.cc:1398] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:06.649276 3145 tablet_bootstrap.cc:492] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:06.649780 3147 ts_tablet_manager.cc:1398] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:12:06.650094 3147 tablet_bootstrap.cc:492] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:06.651819 3145 tablet_bootstrap.cc:654] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:06.656858 3145 tablet_bootstrap.cc:492] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:06.657034 3145 ts_tablet_manager.cc:1367] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 08:12:06.657850 3145 raft_consensus.cc:348] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.658068 3145 raft_consensus.cc:374] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:06.658150 3145 raft_consensus.cc:729] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:06.658416 3145 consensus_queue.cc:260] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.660602 3145 ts_tablet_manager.cc:1398] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:06.660621 3147 tablet_bootstrap.cc:654] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:06.661026 3146 tablet_bootstrap.cc:492] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:06.661213 3146 ts_tablet_manager.cc:1367] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.011s
I20240503 08:12:06.663283 3150 tablet_bootstrap.cc:492] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:06.663242 3146 raft_consensus.cc:348] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.663537 3150 ts_tablet_manager.cc:1367] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.006s
I20240503 08:12:06.663550 3146 raft_consensus.cc:374] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:06.663622 3146 raft_consensus.cc:729] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:06.664044 3147 tablet_bootstrap.cc:492] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:06.664227 3147 ts_tablet_manager.cc:1367] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240503 08:12:06.664368 3150 raft_consensus.cc:348] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.665040 3147 raft_consensus.cc:348] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.665226 3150 raft_consensus.cc:374] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:06.665280 3147 raft_consensus.cc:374] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:06.665344 3150 raft_consensus.cc:729] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:06.665374 3147 raft_consensus.cc:729] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:06.665659 3147 consensus_queue.cc:260] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.665668 3150 consensus_queue.cc:260] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.666832 3147 ts_tablet_manager.cc:1398] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:06.667011 3150 ts_tablet_manager.cc:1398] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:06.672389 3024 raft_consensus.cc:482] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:06.672570 3024 raft_consensus.cc:504] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.673532 3024 leader_election.cc:288] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:06.674039 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5408b903cd14f3384a68a12fe4abcbb" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:06.674259 1813 raft_consensus.cc:2454] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:06.674711 1879 leader_election.cc:302] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:06.675287 3024 raft_consensus.cc:2781] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:06.675488 3024 raft_consensus.cc:482] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.675614 3024 raft_consensus.cc:3037] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:06.675552 3146 consensus_queue.cc:260] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.676256 3146 ts_tablet_manager.cc:1398] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 08:12:06.679625 3024 raft_consensus.cc:504] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.680269 2832 raft_consensus.cc:482] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:06.680449 2832 raft_consensus.cc:504] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.681154 2832 leader_election.cc:288] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:06.681602 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5408b903cd14f3384a68a12fe4abcbb" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:06.681789 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7890ae8e9d7d4b4ba072de7969c20fb0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:06.681867 1813 raft_consensus.cc:3037] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:06.681998 2073 raft_consensus.cc:2454] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:06.682448 2138 leader_election.cc:302] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:06.683672 2832 raft_consensus.cc:2781] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:06.683780 2832 raft_consensus.cc:482] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.683869 2832 raft_consensus.cc:3037] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:06.685191 1813 raft_consensus.cc:2454] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:06.685806 1879 leader_election.cc:302] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:06.686664 2832 raft_consensus.cc:504] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.687194 3024 leader_election.cc:288] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:06.687492 3024 raft_consensus.cc:2781] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:06.687649 3024 raft_consensus.cc:686] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:06.687839 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7890ae8e9d7d4b4ba072de7969c20fb0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:06.688055 2073 raft_consensus.cc:3037] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:06.688036 3024 consensus_queue.cc:237] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.691109 2073 raft_consensus.cc:2454] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:06.691705 2138 leader_election.cc:302] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:06.694176 2832 leader_election.cc:288] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:06.694561 2832 raft_consensus.cc:2781] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:06.694967 2832 raft_consensus.cc:686] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:06.695613 2832 consensus_queue.cc:237] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:06.701335 1663 catalog_manager.cc:5484] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:06.705813 1666 catalog_manager.cc:5484] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:06.707240 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5408b903cd14f3384a68a12fe4abcbb" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:06.707547 2072 raft_consensus.cc:3037] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:06.707231 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5408b903cd14f3384a68a12fe4abcbb" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:06.709821 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7890ae8e9d7d4b4ba072de7969c20fb0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:06.709998 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7890ae8e9d7d4b4ba072de7969c20fb0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:06.710143 1813 raft_consensus.cc:2454] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:06.710486 2072 raft_consensus.cc:2454] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:07.013123 3024 raft_consensus.cc:482] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:07.013338 3024 raft_consensus.cc:504] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:07.014144 3024 leader_election.cc:288] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:07.014861 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e53aef15cc4b80b1793a6a10eb5b2b" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:07.014861 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e53aef15cc4b80b1793a6a10eb5b2b" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:07.015106 1812 raft_consensus.cc:2454] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:07.015127 2204 raft_consensus.cc:2454] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:07.015620 1879 leader_election.cc:302] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:07.016000 3024 raft_consensus.cc:2781] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:07.016113 3024 raft_consensus.cc:482] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.016230 3024 raft_consensus.cc:3037] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:07.018059 3024 raft_consensus.cc:504] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:07.018738 3024 leader_election.cc:288] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:07.019349 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e53aef15cc4b80b1793a6a10eb5b2b" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:07.019398 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23e53aef15cc4b80b1793a6a10eb5b2b" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:07.019573 2204 raft_consensus.cc:3037] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:07.019655 1812 raft_consensus.cc:3037] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:07.022094 2204 raft_consensus.cc:2454] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:07.022137 1812 raft_consensus.cc:2454] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:07.022562 1879 leader_election.cc:302] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:07.022926 3024 raft_consensus.cc:2781] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:07.023129 3024 raft_consensus.cc:686] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:07.023442 3024 consensus_queue.cc:237] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:07.026363 1666 catalog_manager.cc:5484] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:07.104604 3024 consensus_queue.cc:1035] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:07.110394 3160 consensus_queue.cc:1035] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:07.121718 2832 consensus_queue.cc:1035] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:07.128196 1812 raft_consensus.cc:3037] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:07.131613 3162 consensus_queue.cc:1035] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:07.411362 3160 consensus_queue.cc:1035] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:07.417485 3177 consensus_queue.cc:1035] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:07.443243 3166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:07.443686 3166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:07.451597 3166 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 202.35 ms
time per row: 0.101175 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:12:08.780364 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:08.790237 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 044d8e5a69444865980ffc06a6be9080 (DEFAULT_TABLE table=rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.790390 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 044d8e5a69444865980ffc06a6be9080 (DEFAULT_TABLE table=rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.790791 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 044d8e5a69444865980ffc06a6be9080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.790791 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 044d8e5a69444865980ffc06a6be9080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.792282 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 48f411ec0bfc4a318b661441c49bd6ec (DEFAULT_TABLE table=rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.792693 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f411ec0bfc4a318b661441c49bd6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.797389 1792 tablet_service.cc:1467] Processing CreateTablet for tablet dc342fed6c6343a080ae0ff2c53cb140 (DEFAULT_TABLE table=rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.798002 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc342fed6c6343a080ae0ff2c53cb140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.798034 1922 tablet_service.cc:1467] Processing CreateTablet for tablet dc342fed6c6343a080ae0ff2c53cb140 (DEFAULT_TABLE table=rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.798498 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc342fed6c6343a080ae0ff2c53cb140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.798604 3231 tablet_bootstrap.cc:492] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:08.800062 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 044d8e5a69444865980ffc06a6be9080 (DEFAULT_TABLE table=rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.800197 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 48f411ec0bfc4a318b661441c49bd6ec (DEFAULT_TABLE table=rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.800539 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 044d8e5a69444865980ffc06a6be9080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.800869 2182 tablet_service.cc:1467] Processing CreateTablet for tablet dc342fed6c6343a080ae0ff2c53cb140 (DEFAULT_TABLE table=rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.801245 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc342fed6c6343a080ae0ff2c53cb140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.804397 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f411ec0bfc4a318b661441c49bd6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.804307 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 48f411ec0bfc4a318b661441c49bd6ec (DEFAULT_TABLE table=rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:08.804774 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f411ec0bfc4a318b661441c49bd6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:08.814882 3231 tablet_bootstrap.cc:654] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.816581 3232 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:08.818044 3234 tablet_bootstrap.cc:492] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:08.818454 3232 tablet_bootstrap.cc:654] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.819155 3233 tablet_bootstrap.cc:492] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:08.821868 3233 tablet_bootstrap.cc:654] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.822935 3234 tablet_bootstrap.cc:654] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.823901 3232 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:08.824131 3232 ts_tablet_manager.cc:1367] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20240503 08:12:08.824430 3233 tablet_bootstrap.cc:492] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:08.824491 3231 tablet_bootstrap.cc:492] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:08.824654 3231 ts_tablet_manager.cc:1367] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20240503 08:12:08.824673 3233 ts_tablet_manager.cc:1367] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 08:12:08.825135 3232 raft_consensus.cc:348] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.825361 3232 raft_consensus.cc:374] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.825456 3232 raft_consensus.cc:729] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.825462 3231 raft_consensus.cc:348] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:08.825534 3233 raft_consensus.cc:348] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:08.825700 3231 raft_consensus.cc:374] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.825719 3233 raft_consensus.cc:374] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.825811 3231 raft_consensus.cc:729] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.825807 3233 raft_consensus.cc:729] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.825742 3232 consensus_queue.cc:260] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.826037 3231 consensus_queue.cc:260] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:08.826611 3231 ts_tablet_manager.cc:1398] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:08.826753 3232 ts_tablet_manager.cc:1398] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:08.826905 3231 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:08.827104 3232 tablet_bootstrap.cc:492] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:08.828740 3231 tablet_bootstrap.cc:654] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.829406 3232 tablet_bootstrap.cc:654] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.832989 3231 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:08.826064 3233 consensus_queue.cc:260] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:08.833261 3231 ts_tablet_manager.cc:1367] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:12:08.833952 3233 ts_tablet_manager.cc:1398] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:12:08.834110 3231 raft_consensus.cc:348] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.834358 3233 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:08.834450 3231 raft_consensus.cc:374] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.834538 3231 raft_consensus.cc:729] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.834898 3231 consensus_queue.cc:260] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.835887 3231 ts_tablet_manager.cc:1398] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:08.836191 3231 tablet_bootstrap.cc:492] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:08.838346 3231 tablet_bootstrap.cc:654] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.840265 3233 tablet_bootstrap.cc:654] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.843111 3234 tablet_bootstrap.cc:492] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:08.843384 3234 ts_tablet_manager.cc:1367] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.000s
I20240503 08:12:08.844477 3234 raft_consensus.cc:348] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.844774 3234 raft_consensus.cc:374] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.844866 3234 raft_consensus.cc:729] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.845250 3234 consensus_queue.cc:260] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.849354 3231 tablet_bootstrap.cc:492] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:08.849615 3231 ts_tablet_manager.cc:1367] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240503 08:12:08.850775 3231 raft_consensus.cc:348] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.851110 3231 raft_consensus.cc:374] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.851240 3231 raft_consensus.cc:729] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.851657 3231 consensus_queue.cc:260] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.852475 3234 ts_tablet_manager.cc:1398] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:12:08.852665 3231 ts_tablet_manager.cc:1398] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20240503 08:12:08.858124 3232 tablet_bootstrap.cc:492] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:08.858198 3233 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:08.858345 3232 ts_tablet_manager.cc:1367] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.008s
I20240503 08:12:08.858382 3233 ts_tablet_manager.cc:1367] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.011s
I20240503 08:12:08.858984 3232 raft_consensus.cc:348] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:08.858984 3233 raft_consensus.cc:348] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.859174 3233 raft_consensus.cc:374] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.859189 3232 raft_consensus.cc:374] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.859253 3233 raft_consensus.cc:729] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.859264 3232 raft_consensus.cc:729] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.859524 3232 consensus_queue.cc:260] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:08.859553 3233 consensus_queue.cc:260] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.860052 3233 ts_tablet_manager.cc:1398] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:08.860386 3232 ts_tablet_manager.cc:1398] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:08.860455 3233 tablet_bootstrap.cc:492] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:08.862512 3233 tablet_bootstrap.cc:654] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:08.866379 3233 tablet_bootstrap.cc:492] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:08.866559 3233 ts_tablet_manager.cc:1367] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240503 08:12:08.867347 3233 raft_consensus.cc:348] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.867539 3233 raft_consensus.cc:374] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:08.867641 3233 raft_consensus.cc:729] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:08.867836 3233 consensus_queue.cc:260] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.868273 3233 ts_tablet_manager.cc:1398] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:08.872828 2958 raft_consensus.cc:482] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:08.872995 2958 raft_consensus.cc:504] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.873636 2958 leader_election.cc:288] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:08.874181 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044d8e5a69444865980ffc06a6be9080" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:08.874181 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044d8e5a69444865980ffc06a6be9080" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:08.874423 2204 raft_consensus.cc:2454] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:08.874444 1943 raft_consensus.cc:2454] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:08.874992 2008 leader_election.cc:302] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:08.875344 2958 raft_consensus.cc:2781] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:08.875474 2958 raft_consensus.cc:482] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.875579 2958 raft_consensus.cc:3037] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.877622 2958 raft_consensus.cc:504] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.878213 2958 leader_election.cc:288] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:08.878878 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044d8e5a69444865980ffc06a6be9080" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:08.879051 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044d8e5a69444865980ffc06a6be9080" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:08.879132 1943 raft_consensus.cc:3037] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.879274 2204 raft_consensus.cc:3037] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:08.881554 1943 raft_consensus.cc:2454] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:08.881569 2204 raft_consensus.cc:2454] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:08.882054 2009 leader_election.cc:302] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:08.882416 2958 raft_consensus.cc:2781] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:08.882557 2958 raft_consensus.cc:686] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:08.882882 2958 consensus_queue.cc:237] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:08.885891 1665 catalog_manager.cc:5484] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:09.005729 2958 raft_consensus.cc:482] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:09.005959 2958 raft_consensus.cc:504] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:09.006659 2958 leader_election.cc:288] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:09.007355 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f411ec0bfc4a318b661441c49bd6ec" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:09.007355 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f411ec0bfc4a318b661441c49bd6ec" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:09.007604 1943 raft_consensus.cc:2454] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:09.007643 2204 raft_consensus.cc:2454] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:09.008050 2008 leader_election.cc:302] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:09.008371 2958 raft_consensus.cc:2781] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:09.008466 2958 raft_consensus.cc:482] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.008756 2958 raft_consensus.cc:3037] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.010669 2958 raft_consensus.cc:504] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:09.011287 2958 leader_election.cc:288] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:09.011986 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f411ec0bfc4a318b661441c49bd6ec" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:09.011986 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f411ec0bfc4a318b661441c49bd6ec" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:09.012229 2204 raft_consensus.cc:3037] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.012260 1943 raft_consensus.cc:3037] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.015272 2204 raft_consensus.cc:2454] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:09.015272 1943 raft_consensus.cc:2454] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:09.015859 2009 leader_election.cc:302] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:09.016209 2958 raft_consensus.cc:2781] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:09.016404 2958 raft_consensus.cc:686] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:09.016744 2958 consensus_queue.cc:237] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:09.019858 1665 catalog_manager.cc:5484] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:09.137130 2832 raft_consensus.cc:482] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:09.137342 2832 raft_consensus.cc:504] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:09.138077 2832 leader_election.cc:288] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:09.138664 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc342fed6c6343a080ae0ff2c53cb140" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:09.138763 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc342fed6c6343a080ae0ff2c53cb140" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:09.138886 1812 raft_consensus.cc:2454] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:09.139032 1943 raft_consensus.cc:2454] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:09.139421 2139 leader_election.cc:302] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:09.139788 2832 raft_consensus.cc:2781] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:09.139909 2832 raft_consensus.cc:482] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.140019 2832 raft_consensus.cc:3037] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.143081 2832 raft_consensus.cc:504] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:09.144429 2832 leader_election.cc:288] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:09.145226 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc342fed6c6343a080ae0ff2c53cb140" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:09.145318 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc342fed6c6343a080ae0ff2c53cb140" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:09.145509 1943 raft_consensus.cc:3037] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.145507 1812 raft_consensus.cc:3037] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:09.148707 1943 raft_consensus.cc:2454] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:09.148869 1812 raft_consensus.cc:2454] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:09.149323 2139 leader_election.cc:302] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:09.149637 2832 raft_consensus.cc:2781] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:09.150111 2832 raft_consensus.cc:686] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:09.150424 2832 consensus_queue.cc:237] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:09.153980 1665 catalog_manager.cc:5484] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:09.342869 2958 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:09.349370 3251 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:09.546093 3248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:09.546459 3248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:09.554219 3248 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:09.579739 3251 consensus_queue.cc:1035] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:09.584723 3246 consensus_queue.cc:1035] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:09.585230 3226 consensus_queue.cc:1035] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240503 08:12:09.597090 2832 consensus_queue.cc:1035] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 242.355 ms
time per row: 0.121177 ms
I20240503 08:12:09.965775 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:09.975276 1792 tablet_service.cc:1467] Processing CreateTablet for tablet f5fc5f680f1e43cf8083a558b14e4461 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.975595 2054 tablet_service.cc:1467] Processing CreateTablet for tablet f5fc5f680f1e43cf8083a558b14e4461 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.975816 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fc5f680f1e43cf8083a558b14e4461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.976052 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fc5f680f1e43cf8083a558b14e4461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.976661 2052 tablet_service.cc:1467] Processing CreateTablet for tablet a04d01412b4f420884570e24c9b204b8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.977064 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04d01412b4f420884570e24c9b204b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.977452 1922 tablet_service.cc:1467] Processing CreateTablet for tablet f5fc5f680f1e43cf8083a558b14e4461 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.977905 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fc5f680f1e43cf8083a558b14e4461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.978878 2184 tablet_service.cc:1467] Processing CreateTablet for tablet a04d01412b4f420884570e24c9b204b8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.979283 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 657d9622b1a241a0bed20e3474a0093a (DEFAULT_TABLE table=rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.979712 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657d9622b1a241a0bed20e3474a0093a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.979810 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04d01412b4f420884570e24c9b204b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.984431 1793 tablet_service.cc:1467] Processing CreateTablet for tablet a04d01412b4f420884570e24c9b204b8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.984436 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 657d9622b1a241a0bed20e3474a0093a (DEFAULT_TABLE table=rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.984899 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04d01412b4f420884570e24c9b204b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.984903 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657d9622b1a241a0bed20e3474a0093a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.987771 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 657d9622b1a241a0bed20e3474a0093a (DEFAULT_TABLE table=rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:09.988224 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657d9622b1a241a0bed20e3474a0093a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:09.997851 3312 tablet_bootstrap.cc:492] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:09.997924 3311 tablet_bootstrap.cc:492] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:10.000164 3311 tablet_bootstrap.cc:654] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.002122 3313 tablet_bootstrap.cc:492] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:10.004113 3313 tablet_bootstrap.cc:654] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.004112 3312 tablet_bootstrap.cc:654] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.004480 3310 tablet_bootstrap.cc:492] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:10.006305 3311 tablet_bootstrap.cc:492] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:10.006536 3311 ts_tablet_manager.cc:1367] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 08:12:10.006680 3310 tablet_bootstrap.cc:654] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.006778 3313 tablet_bootstrap.cc:492] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:10.007035 3313 ts_tablet_manager.cc:1367] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240503 08:12:10.007624 3311 raft_consensus.cc:348] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:10.007961 3311 raft_consensus.cc:374] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.008118 3311 raft_consensus.cc:729] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.008141 3313 raft_consensus.cc:348] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:10.008569 3313 raft_consensus.cc:374] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.008446 3311 consensus_queue.cc:260] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:10.008654 3312 tablet_bootstrap.cc:492] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:10.008754 3313 raft_consensus.cc:729] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.008944 3312 ts_tablet_manager.cc:1367] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:12:10.009136 3313 consensus_queue.cc:260] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:10.009979 3311 ts_tablet_manager.cc:1398] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:10.010453 3313 ts_tablet_manager.cc:1398] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:10.010704 3313 tablet_bootstrap.cc:492] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:10.012248 3313 tablet_bootstrap.cc:654] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.013254 3312 raft_consensus.cc:348] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.013528 3312 raft_consensus.cc:374] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.013653 3312 raft_consensus.cc:729] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.013964 3312 consensus_queue.cc:260] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.015020 3312 ts_tablet_manager.cc:1398] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:10.015409 3312 tablet_bootstrap.cc:492] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:10.017279 3310 tablet_bootstrap.cc:492] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:10.017493 3310 ts_tablet_manager.cc:1367] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240503 08:12:10.017830 3312 tablet_bootstrap.cc:654] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.018388 3310 raft_consensus.cc:348] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.018648 3310 raft_consensus.cc:374] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.018771 3310 raft_consensus.cc:729] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.019039 3310 consensus_queue.cc:260] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.020654 3310 ts_tablet_manager.cc:1398] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:10.022138 3310 tablet_bootstrap.cc:492] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:10.024963 3310 tablet_bootstrap.cc:654] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.031822 3310 tablet_bootstrap.cc:492] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:10.031860 3312 tablet_bootstrap.cc:492] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:10.031950 3313 tablet_bootstrap.cc:492] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:10.032042 3310 ts_tablet_manager.cc:1367] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20240503 08:12:10.032053 3312 ts_tablet_manager.cc:1367] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20240503 08:12:10.032191 3313 ts_tablet_manager.cc:1367] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.009s
I20240503 08:12:10.032775 3310 raft_consensus.cc:348] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.032984 3310 raft_consensus.cc:374] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.032932 3313 raft_consensus.cc:348] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.032932 3312 raft_consensus.cc:348] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:10.033097 3310 raft_consensus.cc:729] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.033178 3312 raft_consensus.cc:374] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.033178 3313 raft_consensus.cc:374] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.033268 3313 raft_consensus.cc:729] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.033268 3312 raft_consensus.cc:729] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.033411 3310 consensus_queue.cc:260] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.033550 3312 consensus_queue.cc:260] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:10.033600 3313 consensus_queue.cc:260] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.034175 3312 ts_tablet_manager.cc:1398] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:10.034489 3312 tablet_bootstrap.cc:492] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:10.034930 3313 ts_tablet_manager.cc:1398] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:10.035284 3313 tablet_bootstrap.cc:492] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:10.035567 3310 ts_tablet_manager.cc:1398] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:10.036293 3312 tablet_bootstrap.cc:654] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.037160 3313 tablet_bootstrap.cc:654] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:10.039502 3313 tablet_bootstrap.cc:492] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:10.039676 3313 ts_tablet_manager.cc:1367] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:12:10.039819 3312 tablet_bootstrap.cc:492] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:10.039965 3312 ts_tablet_manager.cc:1367] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240503 08:12:10.040527 3313 raft_consensus.cc:348] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.040760 3313 raft_consensus.cc:374] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.040844 3313 raft_consensus.cc:729] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.040730 3312 raft_consensus.cc:348] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.040938 3312 raft_consensus.cc:374] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:10.041072 3312 raft_consensus.cc:729] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:10.041102 3313 consensus_queue.cc:260] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.041373 3312 consensus_queue.cc:260] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.041880 3313 ts_tablet_manager.cc:1398] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:10.042100 3312 ts_tablet_manager.cc:1398] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:10.151955 2958 raft_consensus.cc:482] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:10.152240 2958 raft_consensus.cc:504] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:10.152948 2958 leader_election.cc:288] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:10.153568 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04d01412b4f420884570e24c9b204b8" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:10.153685 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04d01412b4f420884570e24c9b204b8" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:10.153878 2203 raft_consensus.cc:2454] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:10.153923 1812 raft_consensus.cc:2454] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:10.154446 2008 leader_election.cc:302] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:10.154811 2958 raft_consensus.cc:2781] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:10.154917 2958 raft_consensus.cc:482] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.155027 2958 raft_consensus.cc:3037] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.157990 2958 raft_consensus.cc:504] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:10.159346 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04d01412b4f420884570e24c9b204b8" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:10.159350 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04d01412b4f420884570e24c9b204b8" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:10.159602 2203 raft_consensus.cc:3037] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.159610 1812 raft_consensus.cc:3037] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.162678 1812 raft_consensus.cc:2454] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:10.163084 2203 raft_consensus.cc:2454] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:10.163350 2009 leader_election.cc:302] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:10.163758 3251 raft_consensus.cc:2781] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:10.163895 3251 raft_consensus.cc:686] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:10.164227 3251 consensus_queue.cc:237] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:10.164443 2958 leader_election.cc:288] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:10.167762 1665 catalog_manager.cc:5484] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:10.269541 3024 raft_consensus.cc:482] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:10.269764 3024 raft_consensus.cc:504] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.270460 3024 leader_election.cc:288] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:10.270985 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657d9622b1a241a0bed20e3474a0093a" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:10.271112 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657d9622b1a241a0bed20e3474a0093a" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:10.271219 2072 raft_consensus.cc:2454] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:10.271379 1812 raft_consensus.cc:2454] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:10.271750 1878 leader_election.cc:302] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:10.272075 3024 raft_consensus.cc:2781] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:10.272164 3024 raft_consensus.cc:482] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.272245 3024 raft_consensus.cc:3037] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.274184 3024 raft_consensus.cc:504] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.274783 3024 leader_election.cc:288] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:10.275370 3251 raft_consensus.cc:482] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:10.275394 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657d9622b1a241a0bed20e3474a0093a" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:10.275398 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657d9622b1a241a0bed20e3474a0093a" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:10.275604 1812 raft_consensus.cc:3037] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.275663 2072 raft_consensus.cc:3037] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.275604 3251 raft_consensus.cc:504] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.276366 3251 leader_election.cc:288] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:10.276981 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fc5f680f1e43cf8083a558b14e4461" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:10.277216 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fc5f680f1e43cf8083a558b14e4461" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:10.277266 1813 raft_consensus.cc:2454] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:10.277415 1944 raft_consensus.cc:2454] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:10.277942 2009 leader_election.cc:302] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:10.278475 3251 raft_consensus.cc:2781] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:10.278525 2072 raft_consensus.cc:2454] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:10.278594 3251 raft_consensus.cc:482] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.278591 1812 raft_consensus.cc:2454] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:10.278704 3251 raft_consensus.cc:3037] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.279036 1878 leader_election.cc:302] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:10.279354 3024 raft_consensus.cc:2781] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:10.279835 3024 raft_consensus.cc:686] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:10.280437 3024 consensus_queue.cc:237] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.281333 3251 raft_consensus.cc:504] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.282215 3251 leader_election.cc:288] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:10.282703 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fc5f680f1e43cf8083a558b14e4461" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:10.282753 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fc5f680f1e43cf8083a558b14e4461" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:10.282909 1944 raft_consensus.cc:3037] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.283039 1812 raft_consensus.cc:3037] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:10.285389 1944 raft_consensus.cc:2454] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:10.285918 2009 leader_election.cc:302] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:10.286209 3251 raft_consensus.cc:2781] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:10.286401 3251 raft_consensus.cc:686] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:10.286682 3251 consensus_queue.cc:237] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:10.286749 1812 raft_consensus.cc:2454] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:10.291908 1665 catalog_manager.cc:5484] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:10.294100 1664 catalog_manager.cc:5484] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:10.636920 3251 consensus_queue.cc:1035] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:10.645125 2958 consensus_queue.cc:1035] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:10.702261 2958 consensus_queue.cc:1035] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:10.707273 3226 consensus_queue.cc:1035] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:10.732347 3024 consensus_queue.cc:1035] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:10.737484 3024 consensus_queue.cc:1035] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:10.766306 3327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:10.766685 3327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:10.774660 3327 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
W20240503 08:12:11.031649 2008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.196:42183, user_credentials={real_user=slave}} blocked reactor thread for 77129.5us
INSERT report
rows total: 2000
time total: 233.473 ms
time per row: 0.116737 ms
I20240503 08:12:11.172224 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:11.183713 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 733bed28775e4779b74e92d8a395234f (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.184239 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733bed28775e4779b74e92d8a395234f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.184584 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 64973388b1c8466798b93ee424df489d (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.184989 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64973388b1c8466798b93ee424df489d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.185883 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 64973388b1c8466798b93ee424df489d (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.186309 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64973388b1c8466798b93ee424df489d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.187791 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 733bed28775e4779b74e92d8a395234f (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.188230 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733bed28775e4779b74e92d8a395234f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.188830 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 733bed28775e4779b74e92d8a395234f (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.189241 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733bed28775e4779b74e92d8a395234f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.191788 2182 tablet_service.cc:1467] Processing CreateTablet for tablet a752938a2b6c424db0c08bb85b64429b (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.191788 2053 tablet_service.cc:1467] Processing CreateTablet for tablet a752938a2b6c424db0c08bb85b64429b (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.192209 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a752938a2b6c424db0c08bb85b64429b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.192219 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a752938a2b6c424db0c08bb85b64429b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.195839 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 64973388b1c8466798b93ee424df489d (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.196331 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64973388b1c8466798b93ee424df489d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.196436 1792 tablet_service.cc:1467] Processing CreateTablet for tablet a752938a2b6c424db0c08bb85b64429b (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:11.196848 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a752938a2b6c424db0c08bb85b64429b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:11.199064 3387 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:11.202320 3387 tablet_bootstrap.cc:654] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.222304 3390 tablet_bootstrap.cc:492] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:11.222416 3391 tablet_bootstrap.cc:492] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:11.225014 3390 tablet_bootstrap.cc:654] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.227633 3388 tablet_bootstrap.cc:492] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:11.228250 3391 tablet_bootstrap.cc:654] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.231254 3388 tablet_bootstrap.cc:654] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.237761 3387 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:11.237881 3390 tablet_bootstrap.cc:492] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:11.238045 3390 ts_tablet_manager.cc:1367] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20240503 08:12:11.238245 3387 ts_tablet_manager.cc:1367] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.039s user 0.018s sys 0.019s
I20240503 08:12:11.238885 3390 raft_consensus.cc:348] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:11.239104 3390 raft_consensus.cc:374] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.239223 3390 raft_consensus.cc:729] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.239483 3387 raft_consensus.cc:348] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:11.239557 3390 consensus_queue.cc:260] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:11.239673 3391 tablet_bootstrap.cc:492] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:11.239830 3391 ts_tablet_manager.cc:1367] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20240503 08:12:11.239835 3387 raft_consensus.cc:374] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.240022 3387 raft_consensus.cc:729] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.240353 3387 consensus_queue.cc:260] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:11.240669 3391 raft_consensus.cc:348] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:11.240868 3391 raft_consensus.cc:374] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.240954 3391 raft_consensus.cc:729] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.241182 3391 consensus_queue.cc:260] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:11.241973 3387 ts_tablet_manager.cc:1398] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:11.242317 3387 tablet_bootstrap.cc:492] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:11.243172 3390 ts_tablet_manager.cc:1398] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:11.243818 3391 ts_tablet_manager.cc:1398] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:11.244153 3391 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:11.244318 3387 tablet_bootstrap.cc:654] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.244549 3390 tablet_bootstrap.cc:492] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:11.246021 3391 tablet_bootstrap.cc:654] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.246266 3390 tablet_bootstrap.cc:654] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.252864 3387 tablet_bootstrap.cc:492] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:11.252862 3391 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:11.253123 3391 ts_tablet_manager.cc:1367] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 08:12:11.253131 3387 ts_tablet_manager.cc:1367] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20240503 08:12:11.254088 3391 raft_consensus.cc:348] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:11.254338 3391 raft_consensus.cc:374] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.254429 3391 raft_consensus.cc:729] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.254721 3391 consensus_queue.cc:260] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:11.255182 3387 raft_consensus.cc:348] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:11.255476 3387 raft_consensus.cc:374] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.255525 3391 ts_tablet_manager.cc:1398] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:11.255600 3387 raft_consensus.cc:729] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.255877 3387 consensus_queue.cc:260] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:11.256408 3387 ts_tablet_manager.cc:1398] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:11.258101 3388 tablet_bootstrap.cc:492] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:11.258306 3388 ts_tablet_manager.cc:1367] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20240503 08:12:11.259394 3388 raft_consensus.cc:348] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:11.259666 3388 raft_consensus.cc:374] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.259773 3388 raft_consensus.cc:729] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.260057 3388 consensus_queue.cc:260] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:11.262584 3388 ts_tablet_manager.cc:1398] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:11.263065 3388 tablet_bootstrap.cc:492] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:11.265813 3388 tablet_bootstrap.cc:654] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.268680 3390 tablet_bootstrap.cc:492] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:11.268899 3390 ts_tablet_manager.cc:1367] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.011s
I20240503 08:12:11.269361 3388 tablet_bootstrap.cc:492] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:11.269536 3388 ts_tablet_manager.cc:1367] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240503 08:12:11.269800 3390 raft_consensus.cc:348] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:11.270051 3390 raft_consensus.cc:374] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.270180 3390 raft_consensus.cc:729] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.270402 3388 raft_consensus.cc:348] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:11.270460 3390 consensus_queue.cc:260] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:11.270686 3388 raft_consensus.cc:374] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.270777 3388 raft_consensus.cc:729] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.271018 3390 ts_tablet_manager.cc:1398] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:11.271037 3388 consensus_queue.cc:260] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:11.271394 3390 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:11.271644 3388 ts_tablet_manager.cc:1398] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:11.273257 3390 tablet_bootstrap.cc:654] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:11.275298 3390 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:11.275506 3390 ts_tablet_manager.cc:1367] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240503 08:12:11.276165 3390 raft_consensus.cc:348] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:11.276335 3390 raft_consensus.cc:374] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:11.276458 3390 raft_consensus.cc:729] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:11.276708 3390 consensus_queue.cc:260] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:11.277277 3390 ts_tablet_manager.cc:1398] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:11.377264 3226 raft_consensus.cc:482] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:11.377506 3226 raft_consensus.cc:504] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:11.378343 3226 leader_election.cc:288] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:11.378875 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a752938a2b6c424db0c08bb85b64429b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:11.378928 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a752938a2b6c424db0c08bb85b64429b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:11.379137 1812 raft_consensus.cc:2454] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:11.379127 2203 raft_consensus.cc:2454] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:11.379658 2009 leader_election.cc:302] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:11.380128 3226 raft_consensus.cc:2781] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:11.380260 3226 raft_consensus.cc:482] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.380359 3226 raft_consensus.cc:3037] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.382500 3226 raft_consensus.cc:504] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:11.383122 3226 leader_election.cc:288] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:11.383769 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a752938a2b6c424db0c08bb85b64429b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:11.383720 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a752938a2b6c424db0c08bb85b64429b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:11.383997 1812 raft_consensus.cc:3037] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.384016 2203 raft_consensus.cc:3037] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.386976 2203 raft_consensus.cc:2454] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:11.386996 1812 raft_consensus.cc:2454] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:11.387578 2009 leader_election.cc:302] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:11.388070 3226 raft_consensus.cc:2781] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:11.388433 3226 raft_consensus.cc:686] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:11.388631 3024 raft_consensus.cc:482] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:11.388794 3024 raft_consensus.cc:504] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:11.388845 3226 consensus_queue.cc:237] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:11.389590 3024 leader_election.cc:288] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:11.390511 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733bed28775e4779b74e92d8a395234f" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:11.390836 1812 raft_consensus.cc:2454] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:11.391290 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733bed28775e4779b74e92d8a395234f" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:11.391554 2203 raft_consensus.cc:2454] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:11.391644 1879 leader_election.cc:302] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:11.391978 3024 raft_consensus.cc:2781] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:11.392091 3024 raft_consensus.cc:482] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.392192 3024 raft_consensus.cc:3037] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.392769 1665 catalog_manager.cc:5484] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:11.394341 3024 raft_consensus.cc:504] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:11.395028 3024 leader_election.cc:288] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:11.395534 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733bed28775e4779b74e92d8a395234f" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:11.395731 2201 raft_consensus.cc:3037] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.395668 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733bed28775e4779b74e92d8a395234f" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:11.395915 1812 raft_consensus.cc:3037] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.397958 1812 raft_consensus.cc:2454] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:11.397985 2201 raft_consensus.cc:2454] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:11.398418 1879 leader_election.cc:302] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:11.398885 3024 raft_consensus.cc:2781] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:11.399055 3024 raft_consensus.cc:686] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:11.399442 3024 consensus_queue.cc:237] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:11.402904 1665 catalog_manager.cc:5484] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:11.466974 2832 raft_consensus.cc:482] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:11.467311 2832 raft_consensus.cc:504] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:11.468140 2832 leader_election.cc:288] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:11.468724 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64973388b1c8466798b93ee424df489d" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:11.469022 1944 raft_consensus.cc:2454] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:11.468945 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64973388b1c8466798b93ee424df489d" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:11.469192 2072 raft_consensus.cc:2454] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:11.469548 2139 leader_election.cc:302] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:11.469877 2832 raft_consensus.cc:2781] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:11.469998 2832 raft_consensus.cc:482] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.470126 2832 raft_consensus.cc:3037] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.472450 2832 raft_consensus.cc:504] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:11.473217 2832 leader_election.cc:288] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:11.473821 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64973388b1c8466798b93ee424df489d" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:11.473867 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64973388b1c8466798b93ee424df489d" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:11.474082 2072 raft_consensus.cc:3037] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.474100 1944 raft_consensus.cc:3037] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:11.477113 1944 raft_consensus.cc:2454] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:11.477128 2072 raft_consensus.cc:2454] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:11.477823 2138 leader_election.cc:302] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:11.478170 2832 raft_consensus.cc:2781] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:11.478643 2832 raft_consensus.cc:686] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:11.478957 2832 consensus_queue.cc:237] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:11.482239 1665 catalog_manager.cc:5484] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:11.865643 3404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:11.865991 3404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:11.873967 3404 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 08:12:11.941051 2832 consensus_queue.cc:1035] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:11.951197 3428 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:11.980392 3226 consensus_queue.cc:1035] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:11.983450 3374 consensus_queue.cc:1035] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:12.036893 3428 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:12.068678 2832 consensus_queue.cc:1035] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 365.898 ms
time per row: 0.182949 ms
I20240503 08:12:12.391871 1671 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:12.401988 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 7729383eb4b046eca0a4e7ff4c636ea0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.402113 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 7729383eb4b046eca0a4e7ff4c636ea0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.402540 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7729383eb4b046eca0a4e7ff4c636ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.402576 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7729383eb4b046eca0a4e7ff4c636ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.403731 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 310d28bf7a30441b84b1e6bcf06e7619 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.404172 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310d28bf7a30441b84b1e6bcf06e7619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.407828 3478 tablet_bootstrap.cc:492] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:12.407843 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 7729383eb4b046eca0a4e7ff4c636ea0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.407961 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 310d28bf7a30441b84b1e6bcf06e7619 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.408311 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7729383eb4b046eca0a4e7ff4c636ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.408435 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310d28bf7a30441b84b1e6bcf06e7619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.410009 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 595f7bc77944474db016cdb39965765b (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.410459 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595f7bc77944474db016cdb39965765b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.410528 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 310d28bf7a30441b84b1e6bcf06e7619 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.411456 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 595f7bc77944474db016cdb39965765b (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.411867 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595f7bc77944474db016cdb39965765b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.416009 2182 tablet_service.cc:1467] Processing CreateTablet for tablet 595f7bc77944474db016cdb39965765b (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:12.416349 3480 tablet_bootstrap.cc:492] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:12.416481 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595f7bc77944474db016cdb39965765b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.417104 3478 tablet_bootstrap.cc:654] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.418654 3481 tablet_bootstrap.cc:492] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:12.419661 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310d28bf7a30441b84b1e6bcf06e7619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:12.420779 3480 tablet_bootstrap.cc:654] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.421993 3481 tablet_bootstrap.cc:654] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.423067 3479 tablet_bootstrap.cc:492] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:12.425379 3479 tablet_bootstrap.cc:654] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.426916 3481 tablet_bootstrap.cc:492] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:12.427202 3481 ts_tablet_manager.cc:1367] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 08:12:12.428301 3481 raft_consensus.cc:348] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.428699 3481 raft_consensus.cc:374] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.428882 3481 raft_consensus.cc:729] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.429255 3481 consensus_queue.cc:260] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.431756 3480 tablet_bootstrap.cc:492] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:12.431926 3480 ts_tablet_manager.cc:1367] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240503 08:12:12.432760 3480 raft_consensus.cc:348] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:12.432976 3480 raft_consensus.cc:374] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.433090 3480 raft_consensus.cc:729] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.433364 3480 consensus_queue.cc:260] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:12.433471 3481 ts_tablet_manager.cc:1398] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:12.434023 3480 ts_tablet_manager.cc:1398] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:12.453025 3481 tablet_bootstrap.cc:492] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:12.455453 3480 tablet_bootstrap.cc:492] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:12.455715 3481 tablet_bootstrap.cc:654] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.458065 3480 tablet_bootstrap.cc:654] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.464474 3480 tablet_bootstrap.cc:492] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:12.464838 3480 ts_tablet_manager.cc:1367] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 08:12:12.464915 3481 tablet_bootstrap.cc:492] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:12.464934 3478 tablet_bootstrap.cc:492] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:12.464931 3479 tablet_bootstrap.cc:492] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:12.465088 3478 ts_tablet_manager.cc:1367] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.037s
I20240503 08:12:12.465085 3479 ts_tablet_manager.cc:1367] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.018s
I20240503 08:12:12.465212 3481 ts_tablet_manager.cc:1367] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240503 08:12:12.465910 3479 raft_consensus.cc:348] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:12.465886 3480 raft_consensus.cc:348] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.466122 3479 raft_consensus.cc:374] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.466143 3480 raft_consensus.cc:374] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.466041 3478 raft_consensus.cc:348] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.466217 3480 raft_consensus.cc:729] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.466234 3478 raft_consensus.cc:374] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.466285 3479 raft_consensus.cc:729] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.466321 3478 raft_consensus.cc:729] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.466029 3481 raft_consensus.cc:348] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:12.466425 3481 raft_consensus.cc:374] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.466557 3481 raft_consensus.cc:729] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.466547 3479 consensus_queue.cc:260] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:12.466593 3478 consensus_queue.cc:260] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.466945 3481 consensus_queue.cc:260] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:12.467182 3479 ts_tablet_manager.cc:1398] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:12.467569 3479 tablet_bootstrap.cc:492] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:12.467697 3481 ts_tablet_manager.cc:1398] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:12.466737 3480 consensus_queue.cc:260] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.468183 3478 ts_tablet_manager.cc:1398] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:12.468528 3478 tablet_bootstrap.cc:492] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:12.469358 3479 tablet_bootstrap.cc:654] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.470440 3478 tablet_bootstrap.cc:654] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.470710 3480 ts_tablet_manager.cc:1398] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:12.472328 3479 tablet_bootstrap.cc:492] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:12.472826 3481 tablet_bootstrap.cc:492] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:12.472855 3479 ts_tablet_manager.cc:1367] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20240503 08:12:12.474045 3479 raft_consensus.cc:348] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.474418 3479 raft_consensus.cc:374] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.474534 3479 raft_consensus.cc:729] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.474879 3479 consensus_queue.cc:260] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.477365 3481 tablet_bootstrap.cc:654] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:12.479630 3479 ts_tablet_manager.cc:1398] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 08:12:12.482199 3481 tablet_bootstrap.cc:492] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:12.482621 3481 ts_tablet_manager.cc:1367] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240503 08:12:12.482934 3478 tablet_bootstrap.cc:492] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:12.483143 3478 ts_tablet_manager.cc:1367] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20240503 08:12:12.483660 3481 raft_consensus.cc:348] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.483860 3481 raft_consensus.cc:374] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.483953 3481 raft_consensus.cc:729] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.484061 3478 raft_consensus.cc:348] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.484182 3481 consensus_queue.cc:260] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.484277 3478 raft_consensus.cc:374] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:12.484350 3478 raft_consensus.cc:729] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:12.484678 3478 consensus_queue.cc:260] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.484784 3481 ts_tablet_manager.cc:1398] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:12.485277 3478 ts_tablet_manager.cc:1398] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:12.493628 3469 raft_consensus.cc:482] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:12.493851 3469 raft_consensus.cc:504] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.494577 3469 leader_election.cc:288] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:12.495209 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7729383eb4b046eca0a4e7ff4c636ea0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:12.495209 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7729383eb4b046eca0a4e7ff4c636ea0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:12.495467 1812 raft_consensus.cc:2454] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:12.495460 1944 raft_consensus.cc:2454] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:12.496008 2139 leader_election.cc:302] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:12.496085 3449 raft_consensus.cc:482] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:12.496333 3469 raft_consensus.cc:2781] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:12.496307 3449 raft_consensus.cc:504] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.496446 3469 raft_consensus.cc:482] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.496531 3469 raft_consensus.cc:3037] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.497126 3449 leader_election.cc:288] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:12.497771 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595f7bc77944474db016cdb39965765b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:12.498011 1812 raft_consensus.cc:2454] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:12.498467 2009 leader_election.cc:302] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:12.498732 3449 raft_consensus.cc:2781] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:12.498849 3449 raft_consensus.cc:482] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.498944 3449 raft_consensus.cc:3037] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.499164 3469 raft_consensus.cc:504] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.499751 3469 leader_election.cc:288] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:12.500209 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595f7bc77944474db016cdb39965765b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:12.500263 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7729383eb4b046eca0a4e7ff4c636ea0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:12.500451 1944 raft_consensus.cc:3037] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.500460 2201 raft_consensus.cc:2454] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:12.501456 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7729383eb4b046eca0a4e7ff4c636ea0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:12.501670 1812 raft_consensus.cc:3037] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.501747 3449 raft_consensus.cc:504] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.502708 3449 leader_election.cc:288] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:12.502851 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595f7bc77944474db016cdb39965765b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:12.503057 1813 raft_consensus.cc:3037] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.503144 1944 raft_consensus.cc:2454] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:12.503851 2139 leader_election.cc:302] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:12.504225 3469 raft_consensus.cc:2781] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:12.504434 3469 raft_consensus.cc:686] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:12.504673 1812 raft_consensus.cc:2454] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:12.504940 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595f7bc77944474db016cdb39965765b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:12.504801 3469 consensus_queue.cc:237] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.505144 2201 raft_consensus.cc:3037] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.505887 1813 raft_consensus.cc:2454] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:12.507756 2201 raft_consensus.cc:2454] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:12.508020 2009 leader_election.cc:302] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:12.508384 3449 raft_consensus.cc:2781] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:12.508550 3449 raft_consensus.cc:686] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:12.508821 3449 consensus_queue.cc:237] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:12.508818 1665 catalog_manager.cc:5484] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:12.512745 1671 catalog_manager.cc:5484] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:12.547906 3449 raft_consensus.cc:482] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:12.548168 3449 raft_consensus.cc:504] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:12.548836 3449 leader_election.cc:288] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:12.549378 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310d28bf7a30441b84b1e6bcf06e7619" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:12.549434 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310d28bf7a30441b84b1e6bcf06e7619" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:12.549619 1812 raft_consensus.cc:2454] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:12.549696 1944 raft_consensus.cc:2454] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:12.550096 2009 leader_election.cc:302] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:12.550387 3449 raft_consensus.cc:2781] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:12.550501 3449 raft_consensus.cc:482] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.550616 3449 raft_consensus.cc:3037] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.552639 3449 raft_consensus.cc:504] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:12.553265 3449 leader_election.cc:288] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:12.553787 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310d28bf7a30441b84b1e6bcf06e7619" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:12.553969 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310d28bf7a30441b84b1e6bcf06e7619" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:12.554076 1812 raft_consensus.cc:3037] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.554208 1944 raft_consensus.cc:3037] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:12.556203 1812 raft_consensus.cc:2454] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:12.556260 1944 raft_consensus.cc:2454] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:12.556695 2009 leader_election.cc:302] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:12.557046 3449 raft_consensus.cc:2781] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:12.557233 3449 raft_consensus.cc:686] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:12.557819 3449 consensus_queue.cc:237] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:12.561100 1671 catalog_manager.cc:5484] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:12.948582 3494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:12.948999 3494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:12.957309 3494 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 08:12:13.027670 3449 consensus_queue.cc:1035] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:13.083536 3507 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.27.157.254:37795, user_credentials={real_user=slave}} blocked reactor thread for 47353.1us
I20240503 08:12:13.090516 3402 consensus_queue.cc:1035] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:13.096719 3226 consensus_queue.cc:1035] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:13.105836 3402 consensus_queue.cc:1035] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:13.123167 3449 consensus_queue.cc:1035] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:13.132294 3226 consensus_queue.cc:1035] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 355.467 ms
time per row: 0.177734 ms
I20240503 08:12:13.472839 1664 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:13.482316 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb8a87410b047548efdd973bbc86714 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.482515 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb8a87410b047548efdd973bbc86714 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.482815 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb8a87410b047548efdd973bbc86714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.482993 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb8a87410b047548efdd973bbc86714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.483865 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 868bcace806f4d32977b21c876c2362b (DEFAULT_TABLE table=rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.484263 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868bcace806f4d32977b21c876c2362b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.488490 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 7c37418ac97343d1931a8e1239218a38 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.488513 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb8a87410b047548efdd973bbc86714 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.488976 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb8a87410b047548efdd973bbc86714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.489029 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c37418ac97343d1931a8e1239218a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.489209 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 868bcace806f4d32977b21c876c2362b (DEFAULT_TABLE table=rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.489605 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868bcace806f4d32977b21c876c2362b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.492034 2182 tablet_service.cc:1467] Processing CreateTablet for tablet 7c37418ac97343d1931a8e1239218a38 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.492587 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c37418ac97343d1931a8e1239218a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.495956 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 868bcace806f4d32977b21c876c2362b (DEFAULT_TABLE table=rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.496526 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868bcace806f4d32977b21c876c2362b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.499120 3566 tablet_bootstrap.cc:492] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:13.501437 3566 tablet_bootstrap.cc:654] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.514330 3568 tablet_bootstrap.cc:492] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:13.515481 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 7c37418ac97343d1931a8e1239218a38 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:13.516223 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c37418ac97343d1931a8e1239218a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:13.517897 3568 tablet_bootstrap.cc:654] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.528998 3568 tablet_bootstrap.cc:492] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:13.529342 3568 ts_tablet_manager.cc:1367] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20240503 08:12:13.531878 3568 raft_consensus.cc:348] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.532313 3568 raft_consensus.cc:374] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.532507 3568 raft_consensus.cc:729] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.532927 3568 consensus_queue.cc:260] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.534576 3568 ts_tablet_manager.cc:1398] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:13.536612 3570 tablet_bootstrap.cc:492] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:13.541121 3568 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:13.543146 3568 tablet_bootstrap.cc:654] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.544044 3566 tablet_bootstrap.cc:492] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:13.544234 3565 tablet_bootstrap.cc:492] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:13.544334 3566 ts_tablet_manager.cc:1367] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.045s user 0.014s sys 0.013s
I20240503 08:12:13.544802 3570 tablet_bootstrap.cc:654] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.545702 3566 raft_consensus.cc:348] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.546097 3566 raft_consensus.cc:374] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.546348 3565 tablet_bootstrap.cc:654] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.547209 3566 raft_consensus.cc:729] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.548195 3566 consensus_queue.cc:260] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.550571 3566 ts_tablet_manager.cc:1398] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:13.552947 3565 tablet_bootstrap.cc:492] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:13.553272 3565 ts_tablet_manager.cc:1367] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20240503 08:12:13.553469 3568 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:13.553673 3568 ts_tablet_manager.cc:1367] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20240503 08:12:13.554622 3565 raft_consensus.cc:348] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.555027 3565 raft_consensus.cc:374] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.555205 3565 raft_consensus.cc:729] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.555608 3565 consensus_queue.cc:260] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.557785 3565 ts_tablet_manager.cc:1398] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:13.557976 3568 raft_consensus.cc:348] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:13.558254 3568 raft_consensus.cc:374] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.558357 3568 raft_consensus.cc:729] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.558614 3565 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:13.558632 3568 consensus_queue.cc:260] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:13.559247 3568 ts_tablet_manager.cc:1398] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 08:12:13.559734 3568 tablet_bootstrap.cc:492] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:13.560041 3570 tablet_bootstrap.cc:492] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:13.560256 3570 ts_tablet_manager.cc:1367] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.005s
I20240503 08:12:13.561159 3565 tablet_bootstrap.cc:654] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.561275 3570 raft_consensus.cc:348] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.561527 3570 raft_consensus.cc:374] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.561722 3568 tablet_bootstrap.cc:654] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.561731 3570 raft_consensus.cc:729] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.562111 3570 consensus_queue.cc:260] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.562844 3570 ts_tablet_manager.cc:1398] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:13.564491 3570 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:13.566869 3570 tablet_bootstrap.cc:654] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.571604 3565 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:13.571942 3565 ts_tablet_manager.cc:1367] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20240503 08:12:13.573618 3565 raft_consensus.cc:348] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:13.574064 3565 raft_consensus.cc:374] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.574507 3565 raft_consensus.cc:729] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.575434 3565 consensus_queue.cc:260] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:13.577483 3565 ts_tablet_manager.cc:1398] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:13.578095 3565 tablet_bootstrap.cc:492] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:13.581652 3565 tablet_bootstrap.cc:654] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:13.584836 3466 raft_consensus.cc:482] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:13.585023 3570 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:13.585107 3466 raft_consensus.cc:504] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.585283 3570 ts_tablet_manager.cc:1367] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.011s
I20240503 08:12:13.586246 3466 leader_election.cc:288] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:13.586521 3570 raft_consensus.cc:348] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:13.586913 3570 raft_consensus.cc:374] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.587030 3570 raft_consensus.cc:729] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.587440 3570 consensus_queue.cc:260] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:13.588160 3570 ts_tablet_manager.cc:1398] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:13.588238 3565 tablet_bootstrap.cc:492] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:13.588445 3565 ts_tablet_manager.cc:1367] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20240503 08:12:13.589442 3565 raft_consensus.cc:348] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.589656 3565 raft_consensus.cc:374] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.589795 3565 raft_consensus.cc:729] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.590047 3565 consensus_queue.cc:260] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.590704 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868bcace806f4d32977b21c876c2362b" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:13.591015 1942 raft_consensus.cc:2454] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:13.591344 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868bcace806f4d32977b21c876c2362b" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:13.592044 1748 leader_election.cc:302] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:13.590734 3565 ts_tablet_manager.cc:1398] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20240503 08:12:13.592793 1747 leader_election.cc:341] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:13.593740 3568 tablet_bootstrap.cc:492] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:13.594027 3568 ts_tablet_manager.cc:1367] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.025s
I20240503 08:12:13.594620 3466 raft_consensus.cc:2781] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:13.594761 3466 raft_consensus.cc:482] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.594849 3466 raft_consensus.cc:3037] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.595007 3568 raft_consensus.cc:348] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.595228 3568 raft_consensus.cc:374] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:13.595314 3568 raft_consensus.cc:729] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:13.595698 3568 consensus_queue.cc:260] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.596297 3568 ts_tablet_manager.cc:1398] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:13.599743 3466 raft_consensus.cc:504] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.604683 3466 leader_election.cc:288] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:13.605136 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868bcace806f4d32977b21c876c2362b" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:13.605230 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868bcace806f4d32977b21c876c2362b" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:13.605420 1942 raft_consensus.cc:3037] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.605422 2201 raft_consensus.cc:3037] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.608381 2201 raft_consensus.cc:2454] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:13.608397 1942 raft_consensus.cc:2454] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:13.608839 1747 leader_election.cc:302] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:13.609154 3578 raft_consensus.cc:2781] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:13.609287 3578 raft_consensus.cc:686] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:13.609822 3578 consensus_queue.cc:237] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.612946 1667 catalog_manager.cc:5484] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:13.662755 3541 raft_consensus.cc:482] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:13.662984 3541 raft_consensus.cc:504] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.663991 3541 leader_election.cc:288] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:13.664534 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb8a87410b047548efdd973bbc86714" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:13.664860 1942 raft_consensus.cc:2454] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:13.664827 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb8a87410b047548efdd973bbc86714" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:13.665055 2201 raft_consensus.cc:2454] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:13.665566 2009 leader_election.cc:302] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:13.665645 3526 raft_consensus.cc:482] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:13.665817 3526 raft_consensus.cc:504] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:13.666627 3526 leader_election.cc:288] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:13.667491 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c37418ac97343d1931a8e1239218a38" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:13.667793 2072 raft_consensus.cc:2454] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:13.668155 3541 raft_consensus.cc:2781] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:13.668288 3541 raft_consensus.cc:482] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.668291 1878 leader_election.cc:302] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:13.668395 3541 raft_consensus.cc:3037] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.668588 3526 raft_consensus.cc:2781] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:13.668700 3526 raft_consensus.cc:482] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.668799 3526 raft_consensus.cc:3037] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.670848 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c37418ac97343d1931a8e1239218a38" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:13.671084 2201 raft_consensus.cc:2454] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:13.671494 3526 raft_consensus.cc:504] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:13.672151 3526 leader_election.cc:288] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:13.672297 3541 raft_consensus.cc:504] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.673200 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c37418ac97343d1931a8e1239218a38" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:13.673429 2072 raft_consensus.cc:3037] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.673419 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb8a87410b047548efdd973bbc86714" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:13.673481 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb8a87410b047548efdd973bbc86714" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:13.673745 1942 raft_consensus.cc:3037] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.673718 2203 raft_consensus.cc:3037] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.676577 2203 raft_consensus.cc:2454] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:13.676571 2072 raft_consensus.cc:2454] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:13.676901 1942 raft_consensus.cc:2454] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:13.677148 2008 leader_election.cc:302] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:13.677132 1878 leader_election.cc:302] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:13.677517 3449 raft_consensus.cc:2781] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:13.677556 3526 raft_consensus.cc:2781] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:13.677691 3449 raft_consensus.cc:686] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:13.678105 3526 raft_consensus.cc:686] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:13.678000 3449 consensus_queue.cc:237] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:13.678390 3526 consensus_queue.cc:237] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:13.679428 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c37418ac97343d1931a8e1239218a38" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:13.679682 2201 raft_consensus.cc:3037] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:13.683521 3541 leader_election.cc:288] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:13.683279 2201 raft_consensus.cc:2454] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:13.683096 1667 catalog_manager.cc:5484] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:13.684949 1670 catalog_manager.cc:5484] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:14.020982 3578 consensus_queue.cc:1035] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:14.027711 3578 consensus_queue.cc:1035] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:14.060948 3582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:14.061318 3582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:14.069237 3582 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:14.103366 3526 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240503 08:12:14.112304 3428 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:14.259487 3449 consensus_queue.cc:1035] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:14.260275 3449 consensus_queue.cc:1035] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 370.534 ms
time per row: 0.185267 ms
I20240503 08:12:14.608548 1667 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:14.617770 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 660769ce396e4c3ca80ac6c506755c55 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.617964 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 660769ce396e4c3ca80ac6c506755c55 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.618294 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660769ce396e4c3ca80ac6c506755c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.618384 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660769ce396e4c3ca80ac6c506755c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.619397 1923 tablet_service.cc:1467] Processing CreateTablet for tablet af299410e1064415bb06bb045f93659d (DEFAULT_TABLE table=rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.619810 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af299410e1064415bb06bb045f93659d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.619844 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 660769ce396e4c3ca80ac6c506755c55 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.620255 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660769ce396e4c3ca80ac6c506755c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.624585 2052 tablet_service.cc:1467] Processing CreateTablet for tablet af299410e1064415bb06bb045f93659d (DEFAULT_TABLE table=rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.624617 2182 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30093e839f49e6b25b639d24bea0a5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.624976 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af299410e1064415bb06bb045f93659d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.625043 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30093e839f49e6b25b639d24bea0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.627674 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30093e839f49e6b25b639d24bea0a5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.628017 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30093e839f49e6b25b639d24bea0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.630012 3648 tablet_bootstrap.cc:492] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:14.630009 3647 tablet_bootstrap.cc:492] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:14.631790 1791 tablet_service.cc:1467] Processing CreateTablet for tablet af299410e1064415bb06bb045f93659d (DEFAULT_TABLE table=rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.631999 3647 tablet_bootstrap.cc:654] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.632323 3648 tablet_bootstrap.cc:654] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.632359 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af299410e1064415bb06bb045f93659d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.633651 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30093e839f49e6b25b639d24bea0a5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:14.634191 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30093e839f49e6b25b639d24bea0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:14.637408 3648 tablet_bootstrap.cc:492] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:14.637624 3648 ts_tablet_manager.cc:1367] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 08:12:14.638453 3648 raft_consensus.cc:348] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.638788 3648 raft_consensus.cc:374] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.638947 3648 raft_consensus.cc:729] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.639286 3648 consensus_queue.cc:260] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.640136 3648 ts_tablet_manager.cc:1398] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:14.640445 3648 tablet_bootstrap.cc:492] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:14.642329 3648 tablet_bootstrap.cc:654] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.667155 3630 raft_consensus.cc:482] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:14.667423 3630 raft_consensus.cc:504] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.668473 3630 leader_election.cc:288] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:14.669543 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af299410e1064415bb06bb045f93659d" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
W20240503 08:12:14.670792 2009 leader_election.cc:341] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:12:14.671146 2009 leader_election.cc:341] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Not found: Tablet not found: af299410e1064415bb06bb045f93659d
I20240503 08:12:14.671294 2009 leader_election.cc:302] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e
I20240503 08:12:14.671864 3630 raft_consensus.cc:2726] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:14.678601 3653 tablet_bootstrap.cc:492] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:14.680929 3653 tablet_bootstrap.cc:654] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.681772 3648 tablet_bootstrap.cc:492] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:14.681988 3648 ts_tablet_manager.cc:1367] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.025s
I20240503 08:12:14.683269 3648 raft_consensus.cc:348] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:14.683665 3648 raft_consensus.cc:374] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.683779 3648 raft_consensus.cc:729] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.684422 3651 tablet_bootstrap.cc:492] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:14.684674 3648 consensus_queue.cc:260] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:14.685200 3647 tablet_bootstrap.cc:492] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:14.685377 3648 ts_tablet_manager.cc:1398] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:12:14.685405 3647 ts_tablet_manager.cc:1367] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.056s user 0.029s sys 0.013s
I20240503 08:12:14.687088 3651 tablet_bootstrap.cc:654] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.686606 3647 raft_consensus.cc:348] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.687199 3647 raft_consensus.cc:374] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.687343 3647 raft_consensus.cc:729] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.687669 3647 consensus_queue.cc:260] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.691141 3648 tablet_bootstrap.cc:492] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:14.693056 3647 ts_tablet_manager.cc:1398] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 08:12:14.693431 3647 tablet_bootstrap.cc:492] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:14.700096 3648 tablet_bootstrap.cc:654] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.704423 3647 tablet_bootstrap.cc:654] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.705505 3651 tablet_bootstrap.cc:492] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:14.705677 3651 ts_tablet_manager.cc:1367] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.014s
I20240503 08:12:14.706455 3651 raft_consensus.cc:348] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:14.706669 3651 raft_consensus.cc:374] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.706758 3651 raft_consensus.cc:729] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.707010 3651 consensus_queue.cc:260] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:14.707759 3651 ts_tablet_manager.cc:1398] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:14.708352 3653 tablet_bootstrap.cc:492] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:14.708516 3653 ts_tablet_manager.cc:1367] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.011s
I20240503 08:12:14.709113 3648 tablet_bootstrap.cc:492] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:14.709277 3648 ts_tablet_manager.cc:1367] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240503 08:12:14.709359 3653 raft_consensus.cc:348] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.709565 3653 raft_consensus.cc:374] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.709661 3653 raft_consensus.cc:729] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.709885 3653 consensus_queue.cc:260] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.710052 3648 raft_consensus.cc:348] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.710238 3648 raft_consensus.cc:374] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.710326 3648 raft_consensus.cc:729] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.710486 3653 ts_tablet_manager.cc:1398] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 08:12:14.710559 3647 tablet_bootstrap.cc:492] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:14.710554 3648 consensus_queue.cc:260] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.710793 3653 tablet_bootstrap.cc:492] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:14.710774 3647 ts_tablet_manager.cc:1367] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
I20240503 08:12:14.711102 3648 ts_tablet_manager.cc:1398] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:14.711092 3578 raft_consensus.cc:482] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:14.711239 3578 raft_consensus.cc:504] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.711887 3578 leader_election.cc:288] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:14.711875 3647 raft_consensus.cc:348] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.712142 3647 raft_consensus.cc:374] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.712229 3647 raft_consensus.cc:729] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.712486 3647 consensus_queue.cc:260] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.713114 3647 ts_tablet_manager.cc:1398] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:14.715767 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660769ce396e4c3ca80ac6c506755c55" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:14.715814 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660769ce396e4c3ca80ac6c506755c55" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:14.716015 2073 raft_consensus.cc:2454] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:14.716075 1942 raft_consensus.cc:2454] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:14.716533 1747 leader_election.cc:302] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:14.716853 3633 raft_consensus.cc:2781] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:14.716943 3633 raft_consensus.cc:482] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.717026 3633 raft_consensus.cc:3037] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.720106 3633 raft_consensus.cc:504] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.720857 3633 leader_election.cc:288] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:14.721504 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660769ce396e4c3ca80ac6c506755c55" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:14.721527 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660769ce396e4c3ca80ac6c506755c55" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:14.721729 2073 raft_consensus.cc:3037] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.721760 1942 raft_consensus.cc:3037] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.723452 3653 tablet_bootstrap.cc:654] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.724756 1942 raft_consensus.cc:2454] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:14.724982 2073 raft_consensus.cc:2454] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:14.726328 1747 leader_election.cc:302] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:14.726663 3633 raft_consensus.cc:2781] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:14.727167 3633 raft_consensus.cc:686] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:14.727484 3633 consensus_queue.cc:237] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.737565 1667 catalog_manager.cc:5484] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:14.738478 3653 tablet_bootstrap.cc:492] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:14.739434 3653 ts_tablet_manager.cc:1367] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.008s
I20240503 08:12:14.740406 3653 raft_consensus.cc:348] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.740715 3653 raft_consensus.cc:374] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.740795 3653 raft_consensus.cc:729] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.741071 3653 consensus_queue.cc:260] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.741791 3653 ts_tablet_manager.cc:1398] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:14.742175 3653 tablet_bootstrap.cc:492] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:14.743939 3653 tablet_bootstrap.cc:654] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:14.745972 3653 tablet_bootstrap.cc:492] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:14.746126 3653 ts_tablet_manager.cc:1367] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:14.746781 3653 raft_consensus.cc:348] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:14.746953 3653 raft_consensus.cc:374] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:14.747058 3653 raft_consensus.cc:729] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:14.747300 3653 consensus_queue.cc:260] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:14.747884 3653 ts_tablet_manager.cc:1398] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:14.811435 3526 raft_consensus.cc:482] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:14.811689 3526 raft_consensus.cc:504] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.812391 3526 leader_election.cc:288] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:14.812903 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af299410e1064415bb06bb045f93659d" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:14.812964 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af299410e1064415bb06bb045f93659d" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:14.813175 1812 raft_consensus.cc:2454] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:14.813200 2073 raft_consensus.cc:2454] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:14.813647 1879 leader_election.cc:302] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:14.813987 3526 raft_consensus.cc:2781] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:14.814126 3526 raft_consensus.cc:482] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.814227 3526 raft_consensus.cc:3037] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.816392 3526 raft_consensus.cc:504] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.817129 3526 leader_election.cc:288] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:14.817664 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af299410e1064415bb06bb045f93659d" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:14.817683 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af299410e1064415bb06bb045f93659d" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:14.817893 1812 raft_consensus.cc:3037] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.817920 2073 raft_consensus.cc:3037] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:14.820852 2073 raft_consensus.cc:2454] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:14.820864 1812 raft_consensus.cc:2454] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:14.821456 1878 leader_election.cc:302] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:14.821828 3526 raft_consensus.cc:2781] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:14.821972 3526 raft_consensus.cc:686] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:14.822244 3526 consensus_queue.cc:237] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:14.825742 1667 catalog_manager.cc:5484] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:15.058113 3541 raft_consensus.cc:482] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:15.058319 3541 raft_consensus.cc:504] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:15.058950 3541 leader_election.cc:288] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:15.059499 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30093e839f49e6b25b639d24bea0a5" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:15.059536 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30093e839f49e6b25b639d24bea0a5" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:15.059760 1813 raft_consensus.cc:2454] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:15.059747 2201 raft_consensus.cc:2454] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:15.060158 2009 leader_election.cc:302] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:15.060669 3630 raft_consensus.cc:2781] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:15.060770 3630 raft_consensus.cc:482] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.060889 3630 raft_consensus.cc:3037] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:15.063160 3630 raft_consensus.cc:504] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:15.063920 3630 leader_election.cc:288] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:15.064323 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30093e839f49e6b25b639d24bea0a5" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:15.064529 2201 raft_consensus.cc:3037] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:15.064478 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30093e839f49e6b25b639d24bea0a5" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:15.064774 1812 raft_consensus.cc:3037] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:15.066553 2201 raft_consensus.cc:2454] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:15.066558 1812 raft_consensus.cc:2454] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:15.067032 2009 leader_election.cc:302] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:15.067418 3541 raft_consensus.cc:2781] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:15.067632 3541 raft_consensus.cc:686] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:15.067941 3541 consensus_queue.cc:237] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:15.071188 1667 catalog_manager.cc:5484] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:15.158972 3633 consensus_queue.cc:1035] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:15.180395 3578 consensus_queue.cc:1035] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:15.343014 3526 consensus_queue.cc:1035] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:15.363224 3672 consensus_queue.cc:1035] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:15.473362 3541 consensus_queue.cc:1035] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:15.481168 3630 consensus_queue.cc:1035] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:15.520709 3664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:15.521063 3664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:15.529889 3664 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
INSERT report
rows total: 2000
time total: 265.386 ms
time per row: 0.132693 ms
I20240503 08:12:15.953626 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:15.963729 1923 tablet_service.cc:1467] Processing CreateTablet for tablet f3123fdbb5c945bd84f97fb4c35c5523 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:15.964241 2184 tablet_service.cc:1467] Processing CreateTablet for tablet fc4d5ddaaa27463cba31eed19c4c1be0 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:15.964720 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4d5ddaaa27463cba31eed19c4c1be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:15.964835 2182 tablet_service.cc:1467] Processing CreateTablet for tablet f3123fdbb5c945bd84f97fb4c35c5523 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:15.965376 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3123fdbb5c945bd84f97fb4c35c5523. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:15.965368 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3123fdbb5c945bd84f97fb4c35c5523. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:15.965929 1792 tablet_service.cc:1467] Processing CreateTablet for tablet f3123fdbb5c945bd84f97fb4c35c5523 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:15.966516 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3123fdbb5c945bd84f97fb4c35c5523. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:15.973930 2053 tablet_service.cc:1467] Processing CreateTablet for tablet fc4d5ddaaa27463cba31eed19c4c1be0 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:15.974651 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4d5ddaaa27463cba31eed19c4c1be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:15.976084 1793 tablet_service.cc:1467] Processing CreateTablet for tablet fc4d5ddaaa27463cba31eed19c4c1be0 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:15.976672 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4d5ddaaa27463cba31eed19c4c1be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:15.980281 3737 tablet_bootstrap.cc:492] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:15.980320 2183 tablet_service.cc:1467] Processing CreateTablet for tablet fba1e1952da44c0e956be1ef7f592f08 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:15.980567 1922 tablet_service.cc:1467] Processing CreateTablet for tablet fba1e1952da44c0e956be1ef7f592f08 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:15.980928 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba1e1952da44c0e956be1ef7f592f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:15.981078 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba1e1952da44c0e956be1ef7f592f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:15.982264 3737 tablet_bootstrap.cc:654] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:15.989717 2054 tablet_service.cc:1467] Processing CreateTablet for tablet fba1e1952da44c0e956be1ef7f592f08 (DEFAULT_TABLE table=rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:15.990249 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba1e1952da44c0e956be1ef7f592f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:16.014575 3741 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:16.017999 3741 tablet_bootstrap.cc:654] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.019623 3738 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:16.024557 3741 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:16.024761 3741 ts_tablet_manager.cc:1367] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 08:12:16.024899 3738 tablet_bootstrap.cc:654] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.024897 3737 tablet_bootstrap.cc:492] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:16.025224 3737 ts_tablet_manager.cc:1367] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.017s
I20240503 08:12:16.025663 3741 raft_consensus.cc:348] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:16.025951 3741 raft_consensus.cc:374] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.026134 3741 raft_consensus.cc:729] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.026504 3740 tablet_bootstrap.cc:492] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:16.026451 3741 consensus_queue.cc:260] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:16.026731 3737 raft_consensus.cc:348] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:16.026973 3737 raft_consensus.cc:374] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.027570 3741 ts_tablet_manager.cc:1398] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:16.027904 3741 tablet_bootstrap.cc:492] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:16.028033 3737 raft_consensus.cc:729] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.028455 3737 consensus_queue.cc:260] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:16.029834 3741 tablet_bootstrap.cc:654] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.032002 3737 ts_tablet_manager.cc:1398] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 08:12:16.033973 3740 tablet_bootstrap.cc:654] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.035416 3737 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:16.037700 3737 tablet_bootstrap.cc:654] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.042075 3740 tablet_bootstrap.cc:492] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:16.043155 3741 tablet_bootstrap.cc:492] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:16.043418 3741 ts_tablet_manager.cc:1367] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20240503 08:12:16.044173 3738 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:16.044463 3738 ts_tablet_manager.cc:1367] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.000s
I20240503 08:12:16.044358 3741 raft_consensus.cc:348] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:16.044590 3741 raft_consensus.cc:374] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.044685 3741 raft_consensus.cc:729] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.044957 3741 consensus_queue.cc:260] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:16.045733 3738 raft_consensus.cc:348] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:16.046012 3738 raft_consensus.cc:374] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.046087 3741 ts_tablet_manager.cc:1398] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:16.046205 3738 raft_consensus.cc:729] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.046468 3737 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:16.046712 3737 ts_tablet_manager.cc:1367] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240503 08:12:16.047585 3737 raft_consensus.cc:348] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:16.048285 3737 raft_consensus.cc:374] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.048424 3737 raft_consensus.cc:729] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.048427 3738 consensus_queue.cc:260] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:16.048760 3737 consensus_queue.cc:260] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:16.049150 3738 ts_tablet_manager.cc:1398] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:16.049474 3737 ts_tablet_manager.cc:1398] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:16.049628 3738 tablet_bootstrap.cc:492] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:16.043172 3740 ts_tablet_manager.cc:1367] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240503 08:12:16.050627 3740 raft_consensus.cc:348] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:16.051846 3737 tablet_bootstrap.cc:492] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:16.053949 3737 tablet_bootstrap.cc:654] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.053968 3738 tablet_bootstrap.cc:654] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.055028 3740 raft_consensus.cc:374] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.055280 3740 raft_consensus.cc:729] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.055648 3740 consensus_queue.cc:260] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:16.056416 3740 ts_tablet_manager.cc:1398] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:16.056771 3737 tablet_bootstrap.cc:492] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:16.056968 3737 ts_tablet_manager.cc:1367] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 08:12:16.058075 3740 tablet_bootstrap.cc:492] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:16.058326 3737 raft_consensus.cc:348] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:16.058568 3737 raft_consensus.cc:374] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.058717 3737 raft_consensus.cc:729] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.059051 3737 consensus_queue.cc:260] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:16.061264 3740 tablet_bootstrap.cc:654] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:16.061542 3737 ts_tablet_manager.cc:1398] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:16.063479 3738 tablet_bootstrap.cc:492] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:16.063671 3738 ts_tablet_manager.cc:1367] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 08:12:16.064570 3738 raft_consensus.cc:348] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:16.064793 3740 tablet_bootstrap.cc:492] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:16.064991 3740 ts_tablet_manager.cc:1367] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 08:12:16.065130 3738 raft_consensus.cc:374] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.065274 3738 raft_consensus.cc:729] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.065652 3738 consensus_queue.cc:260] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:16.066133 3740 raft_consensus.cc:348] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:16.066388 3738 ts_tablet_manager.cc:1398] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:16.066399 3740 raft_consensus.cc:374] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:16.066476 3740 raft_consensus.cc:729] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:16.066783 3740 consensus_queue.cc:260] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:16.067478 3740 ts_tablet_manager.cc:1398] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:16.083425 3674 raft_consensus.cc:482] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:16.083602 3674 raft_consensus.cc:504] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:16.084378 3674 leader_election.cc:288] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:16.084959 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3123fdbb5c945bd84f97fb4c35c5523" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:16.084966 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3123fdbb5c945bd84f97fb4c35c5523" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:16.085232 2203 raft_consensus.cc:2454] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:16.085232 1813 raft_consensus.cc:2454] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:16.085724 1878 leader_election.cc:302] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:16.086112 3674 raft_consensus.cc:2781] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:16.086249 3674 raft_consensus.cc:482] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.086344 3674 raft_consensus.cc:3037] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.089166 3674 raft_consensus.cc:504] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:16.090466 3674 leader_election.cc:288] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:16.091249 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3123fdbb5c945bd84f97fb4c35c5523" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:16.091411 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3123fdbb5c945bd84f97fb4c35c5523" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:16.091521 1813 raft_consensus.cc:3037] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.091675 2203 raft_consensus.cc:3037] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.093729 1813 raft_consensus.cc:2454] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:16.093981 2203 raft_consensus.cc:2454] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:16.094357 1879 leader_election.cc:302] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:16.094748 3674 raft_consensus.cc:2781] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:16.094929 3674 raft_consensus.cc:686] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:16.095289 3674 consensus_queue.cc:237] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:16.098939 1670 catalog_manager.cc:5484] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:16.209830 3668 raft_consensus.cc:482] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:16.210053 3668 raft_consensus.cc:504] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:16.210732 3668 leader_election.cc:288] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:16.211447 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4d5ddaaa27463cba31eed19c4c1be0" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:16.211580 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4d5ddaaa27463cba31eed19c4c1be0" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:16.211694 2073 raft_consensus.cc:2454] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:16.211823 2203 raft_consensus.cc:2454] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:16.212225 1747 leader_election.cc:302] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:16.212555 3668 raft_consensus.cc:2781] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:16.212711 3668 raft_consensus.cc:482] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.212818 3668 raft_consensus.cc:3037] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.214967 3668 raft_consensus.cc:504] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:16.215682 3668 leader_election.cc:288] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:16.216274 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4d5ddaaa27463cba31eed19c4c1be0" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:16.216351 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4d5ddaaa27463cba31eed19c4c1be0" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:16.216477 2203 raft_consensus.cc:3037] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.216564 2073 raft_consensus.cc:3037] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.218502 2073 raft_consensus.cc:2454] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:16.218508 2203 raft_consensus.cc:2454] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:16.219044 1747 leader_election.cc:302] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:16.219408 3668 raft_consensus.cc:2781] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:16.219559 3668 raft_consensus.cc:686] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:16.219872 3668 consensus_queue.cc:237] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:16.223384 1670 catalog_manager.cc:5484] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:16.292235 3674 raft_consensus.cc:482] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:16.292449 3674 raft_consensus.cc:504] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:16.293148 3674 leader_election.cc:288] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:16.293802 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba1e1952da44c0e956be1ef7f592f08" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:16.293824 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba1e1952da44c0e956be1ef7f592f08" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:16.294057 2203 raft_consensus.cc:2454] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:16.294070 2073 raft_consensus.cc:2454] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:16.294531 1878 leader_election.cc:302] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:16.294881 3674 raft_consensus.cc:2781] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:16.294988 3674 raft_consensus.cc:482] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.295094 3674 raft_consensus.cc:3037] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.296993 3674 raft_consensus.cc:504] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:16.297658 3674 leader_election.cc:288] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:16.298105 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba1e1952da44c0e956be1ef7f592f08" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:16.298177 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba1e1952da44c0e956be1ef7f592f08" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:16.298328 2203 raft_consensus.cc:3037] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.298420 2073 raft_consensus.cc:3037] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:16.300465 2203 raft_consensus.cc:2454] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:16.300493 2073 raft_consensus.cc:2454] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:16.301046 1878 leader_election.cc:302] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:16.301335 3674 raft_consensus.cc:2781] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:16.301506 3674 raft_consensus.cc:686] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:16.301792 3674 consensus_queue.cc:237] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:16.305104 1670 catalog_manager.cc:5484] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:16.556053 3674 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:16.566955 3674 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:16.636870 3668 consensus_queue.cc:1035] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:16.642753 3761 consensus_queue.cc:1035] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:16.731472 3757 consensus_queue.cc:1035] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:16.738004 3674 consensus_queue.cc:1035] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:16.777999 3753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:16.778390 3753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:16.786216 3753 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
INSERT report
rows total: 2000
time total: 283.212 ms
time per row: 0.141606 ms
I20240503 08:12:17.243839 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:17.254127 1923 tablet_service.cc:1467] Processing CreateTablet for tablet de60c691fd194765a969697118c9b90b (DEFAULT_TABLE table=rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:17.254310 2183 tablet_service.cc:1467] Processing CreateTablet for tablet de60c691fd194765a969697118c9b90b (DEFAULT_TABLE table=rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:17.254671 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de60c691fd194765a969697118c9b90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:17.254698 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de60c691fd194765a969697118c9b90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:17.255218 2054 tablet_service.cc:1467] Processing CreateTablet for tablet d3b2ebb6eee649199297eb13faa9e4cb (DEFAULT_TABLE table=rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:17.255807 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b2ebb6eee649199297eb13faa9e4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:17.255775 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 8da7019189fe4b36bbf08cba59a586a5 (DEFAULT_TABLE table=rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:17.256129 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8da7019189fe4b36bbf08cba59a586a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:17.258865 1793 tablet_service.cc:1467] Processing CreateTablet for tablet d3b2ebb6eee649199297eb13faa9e4cb (DEFAULT_TABLE table=rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:17.258867 1922 tablet_service.cc:1467] Processing CreateTablet for tablet d3b2ebb6eee649199297eb13faa9e4cb (DEFAULT_TABLE table=rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:17.259294 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b2ebb6eee649199297eb13faa9e4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:17.259289 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b2ebb6eee649199297eb13faa9e4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:17.260243 1792 tablet_service.cc:1467] Processing CreateTablet for tablet de60c691fd194765a969697118c9b90b (DEFAULT_TABLE table=rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:17.260699 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de60c691fd194765a969697118c9b90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:17.266183 2182 tablet_service.cc:1467] Processing CreateTablet for tablet 8da7019189fe4b36bbf08cba59a586a5 (DEFAULT_TABLE table=rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:17.266687 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8da7019189fe4b36bbf08cba59a586a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:17.266577 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 8da7019189fe4b36bbf08cba59a586a5 (DEFAULT_TABLE table=rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:17.267025 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8da7019189fe4b36bbf08cba59a586a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:17.273283 3823 tablet_bootstrap.cc:492] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:17.275110 3822 tablet_bootstrap.cc:492] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:17.278095 3822 tablet_bootstrap.cc:654] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:17.278752 3823 tablet_bootstrap.cc:654] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:17.280113 3824 tablet_bootstrap.cc:492] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:17.282270 3821 tablet_bootstrap.cc:492] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:17.285312 3822 tablet_bootstrap.cc:492] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:17.285518 3823 tablet_bootstrap.cc:492] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:17.285668 3822 ts_tablet_manager.cc:1367] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 08:12:17.285676 3823 ts_tablet_manager.cc:1367] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20240503 08:12:17.285995 3821 tablet_bootstrap.cc:654] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:17.286417 3822 raft_consensus.cc:348] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:17.286624 3822 raft_consensus.cc:374] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:17.286724 3822 raft_consensus.cc:729] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:17.287178 3823 raft_consensus.cc:348] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:17.287426 3823 raft_consensus.cc:374] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:17.287528 3823 raft_consensus.cc:729] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:17.287853 3824 tablet_bootstrap.cc:654] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:17.287783 3823 consensus_queue.cc:260] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:17.288010 3822 consensus_queue.cc:260] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:17.288776 3821 tablet_bootstrap.cc:492] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:17.288949 3821 ts_tablet_manager.cc:1367] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 08:12:17.289664 3822 ts_tablet_manager.cc:1398] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:17.289985 3822 tablet_bootstrap.cc:492] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:17.291818 3821 raft_consensus.cc:348] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:17.292212 3821 raft_consensus.cc:374] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:17.292380 3821 raft_consensus.cc:729] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:17.292481 3823 ts_tablet_manager.cc:1398] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 08:12:17.292837 3823 tablet_bootstrap.cc:492] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:17.292676 3821 consensus_queue.cc:260] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:17.293478 3821 ts_tablet_manager.cc:1398] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:17.293888 3821 tablet_bootstrap.cc:492] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:17.295778 3823 tablet_bootstrap.cc:654] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:17.296078 3821 tablet_bootstrap.cc:654] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:17.297991 3822 tablet_bootstrap.cc:654] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:17.298612 3824 tablet_bootstrap.cc:492] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:17.298890 3824 ts_tablet_manager.cc:1367] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.002s
I20240503 08:12:17.300015 3824 raft_consensus.cc:348] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.300431 3824 raft_consensus.cc:374] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:17.300611 3824 raft_consensus.cc:729] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:17.301003 3824 consensus_queue.cc:260] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.301769 3824 ts_tablet_manager.cc:1398] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:17.302736 3824 tablet_bootstrap.cc:492] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:17.303215 3821 tablet_bootstrap.cc:492] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:17.303480 3821 ts_tablet_manager.cc:1367] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 08:12:17.304302 3821 raft_consensus.cc:348] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.304543 3821 raft_consensus.cc:374] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:17.304639 3821 raft_consensus.cc:729] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:17.304951 3821 consensus_queue.cc:260] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.305595 3821 ts_tablet_manager.cc:1398] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:17.306008 3821 tablet_bootstrap.cc:492] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:17.306228 3823 tablet_bootstrap.cc:492] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:17.306459 3823 ts_tablet_manager.cc:1367] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.002s
I20240503 08:12:17.307416 3823 raft_consensus.cc:348] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.307732 3823 raft_consensus.cc:374] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:17.307871 3823 raft_consensus.cc:729] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:17.307968 3821 tablet_bootstrap.cc:654] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:17.308223 3823 consensus_queue.cc:260] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.310462 3823 ts_tablet_manager.cc:1398] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:17.311110 3824 tablet_bootstrap.cc:654] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:17.311777 3541 raft_consensus.cc:482] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:17.311987 3541 raft_consensus.cc:504] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:17.312997 3541 leader_election.cc:288] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:17.313906 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b2ebb6eee649199297eb13faa9e4cb" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:17.313897 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b2ebb6eee649199297eb13faa9e4cb" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:17.314141 1942 raft_consensus.cc:2454] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:17.314148 1811 raft_consensus.cc:2454] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:17.315106 2009 leader_election.cc:302] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:17.315943 3541 raft_consensus.cc:2781] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:17.316071 3541 raft_consensus.cc:482] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.316200 3541 raft_consensus.cc:3037] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:17.317025 3821 tablet_bootstrap.cc:492] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:17.317201 3821 ts_tablet_manager.cc:1367] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 08:12:17.317703 3824 tablet_bootstrap.cc:492] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:17.317947 3824 ts_tablet_manager.cc:1367] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.002s
I20240503 08:12:17.318040 3821 raft_consensus.cc:348] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.318331 3822 tablet_bootstrap.cc:492] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:17.318324 3821 raft_consensus.cc:374] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:17.318425 3821 raft_consensus.cc:729] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:17.318493 3822 ts_tablet_manager.cc:1367] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.003s
I20240503 08:12:17.318671 3821 consensus_queue.cc:260] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.319382 3824 raft_consensus.cc:348] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.319375 3822 raft_consensus.cc:348] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.319459 3541 raft_consensus.cc:504] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:17.319691 3824 raft_consensus.cc:374] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:17.319861 3824 raft_consensus.cc:729] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:17.320209 3821 ts_tablet_manager.cc:1398] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:17.320194 3824 consensus_queue.cc:260] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.319625 3822 raft_consensus.cc:374] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:17.320523 3822 raft_consensus.cc:729] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:17.320793 3822 consensus_queue.cc:260] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.320955 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b2ebb6eee649199297eb13faa9e4cb" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:17.321187 1811 raft_consensus.cc:3037] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:17.321358 3822 ts_tablet_manager.cc:1398] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:17.322116 3824 ts_tablet_manager.cc:1398] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:17.324363 1811 raft_consensus.cc:2454] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:17.324987 2009 leader_election.cc:302] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:17.325405 3541 leader_election.cc:288] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:17.325613 3541 raft_consensus.cc:2781] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:17.325784 3541 raft_consensus.cc:686] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:17.326078 3541 consensus_queue.cc:237] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:17.326205 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b2ebb6eee649199297eb13faa9e4cb" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:17.326436 1942 raft_consensus.cc:3037] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:17.328948 1942 raft_consensus.cc:2454] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:17.333365 1671 catalog_manager.cc:5484] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:17.371547 3733 raft_consensus.cc:482] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:17.371722 3733 raft_consensus.cc:504] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.372510 3733 leader_election.cc:288] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:17.373009 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8da7019189fe4b36bbf08cba59a586a5" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:17.373080 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8da7019189fe4b36bbf08cba59a586a5" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:17.373214 2073 raft_consensus.cc:2454] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:17.373345 1811 raft_consensus.cc:2454] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:17.373711 2138 leader_election.cc:302] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:17.374033 3733 raft_consensus.cc:2781] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:17.374138 3733 raft_consensus.cc:482] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.374205 3733 raft_consensus.cc:3037] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:17.376261 3733 raft_consensus.cc:504] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.376849 3733 leader_election.cc:288] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:17.377503 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8da7019189fe4b36bbf08cba59a586a5" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:17.377514 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8da7019189fe4b36bbf08cba59a586a5" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:17.377707 2073 raft_consensus.cc:3037] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:17.377748 1811 raft_consensus.cc:3037] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:17.379830 1811 raft_consensus.cc:2454] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:17.379812 2073 raft_consensus.cc:2454] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:17.380331 2139 leader_election.cc:302] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:17.380685 3733 raft_consensus.cc:2781] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:17.381124 3733 raft_consensus.cc:686] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:17.381475 3733 consensus_queue.cc:237] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.385326 1671 catalog_manager.cc:5484] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:17.574036 3807 raft_consensus.cc:482] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:17.574244 3807 raft_consensus.cc:504] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.575047 3807 leader_election.cc:288] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:17.575516 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de60c691fd194765a969697118c9b90b" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:17.575762 1811 raft_consensus.cc:2454] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:17.575832 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de60c691fd194765a969697118c9b90b" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:17.576081 2202 raft_consensus.cc:2454] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:17.576203 1879 leader_election.cc:302] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:17.576493 3757 raft_consensus.cc:2781] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:17.576658 3757 raft_consensus.cc:482] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.576764 3757 raft_consensus.cc:3037] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:17.579735 3757 raft_consensus.cc:504] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.580309 3757 leader_election.cc:288] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:17.580962 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de60c691fd194765a969697118c9b90b" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:17.581198 2202 raft_consensus.cc:3037] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:17.583252 2202 raft_consensus.cc:2454] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:17.583858 1878 leader_election.cc:302] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:17.583983 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de60c691fd194765a969697118c9b90b" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:17.584194 3757 raft_consensus.cc:2781] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:17.584237 1811 raft_consensus.cc:3037] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:17.584370 3757 raft_consensus.cc:686] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:17.584677 3757 consensus_queue.cc:237] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:17.586740 1811 raft_consensus.cc:2454] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:17.588193 1671 catalog_manager.cc:5484] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:17.903870 3733 consensus_queue.cc:1035] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:17.914103 3733 consensus_queue.cc:1035] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:17.915951 3763 consensus_queue.cc:1035] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:17.928366 3541 consensus_queue.cc:1035] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:18.019488 3757 consensus_queue.cc:1035] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:18.025410 3757 consensus_queue.cc:1035] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:18.031280 3838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:18.031817 3838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:18.041193 3838 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
W20240503 08:12:18.347632 1878 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.196:42183, user_credentials={real_user=slave}} blocked reactor thread for 50707us
INSERT report
rows total: 2000
time total: 358.687 ms
time per row: 0.179343 ms
I20240503 08:12:18.576645 1669 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:18.587118 2182 tablet_service.cc:1467] Processing CreateTablet for tablet a92cb30e806b4e58a2d9f9cfa516bdf9 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.587442 1922 tablet_service.cc:1467] Processing CreateTablet for tablet a92cb30e806b4e58a2d9f9cfa516bdf9 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.587484 2053 tablet_service.cc:1467] Processing CreateTablet for tablet a92cb30e806b4e58a2d9f9cfa516bdf9 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.587911 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92cb30e806b4e58a2d9f9cfa516bdf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.587903 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92cb30e806b4e58a2d9f9cfa516bdf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.587954 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92cb30e806b4e58a2d9f9cfa516bdf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.593361 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 89d9b1b3c23241d49bfcb914a5bd1777 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.593860 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d9b1b3c23241d49bfcb914a5bd1777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.594839 1793 tablet_service.cc:1467] Processing CreateTablet for tablet fa111e798e3f46d2a5bc1eebfe28f8bf (DEFAULT_TABLE table=rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.595613 2184 tablet_service.cc:1467] Processing CreateTablet for tablet fa111e798e3f46d2a5bc1eebfe28f8bf (DEFAULT_TABLE table=rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.595701 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 89d9b1b3c23241d49bfcb914a5bd1777 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.596040 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa111e798e3f46d2a5bc1eebfe28f8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.596315 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d9b1b3c23241d49bfcb914a5bd1777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.597034 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 89d9b1b3c23241d49bfcb914a5bd1777 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.597443 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d9b1b3c23241d49bfcb914a5bd1777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.599675 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa111e798e3f46d2a5bc1eebfe28f8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.600864 1924 tablet_service.cc:1467] Processing CreateTablet for tablet fa111e798e3f46d2a5bc1eebfe28f8bf (DEFAULT_TABLE table=rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:18.601493 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa111e798e3f46d2a5bc1eebfe28f8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:18.605942 3909 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:18.606621 3908 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:18.607867 3909 tablet_bootstrap.cc:654] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.608932 3908 tablet_bootstrap.cc:654] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.611120 3912 tablet_bootstrap.cc:492] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:18.613982 3912 tablet_bootstrap.cc:654] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.625370 3914 tablet_bootstrap.cc:492] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:18.626437 3912 tablet_bootstrap.cc:492] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:18.626626 3912 ts_tablet_manager.cc:1367] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.004s
I20240503 08:12:18.627306 3912 raft_consensus.cc:348] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:18.627813 3912 raft_consensus.cc:374] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.627957 3912 raft_consensus.cc:729] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.628324 3912 consensus_queue.cc:260] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:18.629719 3914 tablet_bootstrap.cc:654] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.630781 3912 ts_tablet_manager.cc:1398] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:18.631291 3912 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:18.633055 3909 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:18.633231 3909 ts_tablet_manager.cc:1367] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.005s
I20240503 08:12:18.634089 3909 raft_consensus.cc:348] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:18.634335 3909 raft_consensus.cc:374] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.634485 3909 raft_consensus.cc:729] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.634756 3909 consensus_queue.cc:260] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:18.636447 3912 tablet_bootstrap.cc:654] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.636934 3909 ts_tablet_manager.cc:1398] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:12:18.637566 3909 tablet_bootstrap.cc:492] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:18.639853 3909 tablet_bootstrap.cc:654] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.640197 3908 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:18.640367 3908 ts_tablet_manager.cc:1367] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.004s
I20240503 08:12:18.641243 3908 raft_consensus.cc:348] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:18.641479 3908 raft_consensus.cc:374] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.641588 3908 raft_consensus.cc:729] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.641857 3908 consensus_queue.cc:260] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:18.642649 3908 ts_tablet_manager.cc:1398] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:18.643584 3908 tablet_bootstrap.cc:492] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:18.643971 3914 tablet_bootstrap.cc:492] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:18.644212 3914 ts_tablet_manager.cc:1367] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 08:12:18.644349 3909 tablet_bootstrap.cc:492] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:18.644531 3909 ts_tablet_manager.cc:1367] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 08:12:18.645256 3914 raft_consensus.cc:348] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:18.645272 3909 raft_consensus.cc:348] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:18.645426 3908 tablet_bootstrap.cc:654] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.645504 3909 raft_consensus.cc:374] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.645570 3909 raft_consensus.cc:729] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.645821 3909 consensus_queue.cc:260] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:18.646399 3909 ts_tablet_manager.cc:1398] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:18.646833 3914 raft_consensus.cc:374] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.647003 3914 raft_consensus.cc:729] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.647670 3914 consensus_queue.cc:260] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:18.649420 3914 ts_tablet_manager.cc:1398] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:18.650815 3908 tablet_bootstrap.cc:492] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:18.651046 3908 ts_tablet_manager.cc:1367] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240503 08:12:18.652503 3912 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:18.652734 3912 ts_tablet_manager.cc:1367] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.007s
I20240503 08:12:18.653584 3912 raft_consensus.cc:348] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:18.653882 3912 raft_consensus.cc:374] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.653985 3912 raft_consensus.cc:729] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.655789 3912 consensus_queue.cc:260] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:18.656517 3912 ts_tablet_manager.cc:1398] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:12:18.658200 3912 tablet_bootstrap.cc:492] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:18.660009 3912 tablet_bootstrap.cc:654] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.661384 3909 tablet_bootstrap.cc:492] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:18.660956 3908 raft_consensus.cc:348] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:18.662413 3908 raft_consensus.cc:374] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.662575 3908 raft_consensus.cc:729] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.662878 3908 consensus_queue.cc:260] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:18.663990 3908 ts_tablet_manager.cc:1398] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240503 08:12:18.665010 3909 tablet_bootstrap.cc:654] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:18.670143 3912 tablet_bootstrap.cc:492] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:18.670464 3912 ts_tablet_manager.cc:1367] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240503 08:12:18.671406 3912 raft_consensus.cc:348] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:18.671705 3912 raft_consensus.cc:374] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.671841 3912 raft_consensus.cc:729] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.672166 3912 consensus_queue.cc:260] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:18.672937 3912 ts_tablet_manager.cc:1398] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:18.674432 3909 tablet_bootstrap.cc:492] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:18.674654 3909 ts_tablet_manager.cc:1367] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20240503 08:12:18.675518 3909 raft_consensus.cc:348] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:18.677362 3909 raft_consensus.cc:374] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:18.677537 3909 raft_consensus.cc:729] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:18.677841 3909 consensus_queue.cc:260] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:18.678473 3909 ts_tablet_manager.cc:1398] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:12:18.694360 3763 raft_consensus.cc:482] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:18.694566 3763 raft_consensus.cc:504] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:18.695415 3763 leader_election.cc:288] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:18.696099 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92cb30e806b4e58a2d9f9cfa516bdf9" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:18.696142 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92cb30e806b4e58a2d9f9cfa516bdf9" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:18.696333 2203 raft_consensus.cc:2454] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:18.696363 1944 raft_consensus.cc:2454] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:18.696833 2008 leader_election.cc:302] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:18.697154 3763 raft_consensus.cc:2781] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:18.697264 3763 raft_consensus.cc:482] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.697362 3763 raft_consensus.cc:3037] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.699288 3763 raft_consensus.cc:504] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:18.700023 3763 leader_election.cc:288] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:18.700562 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92cb30e806b4e58a2d9f9cfa516bdf9" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:18.700615 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a92cb30e806b4e58a2d9f9cfa516bdf9" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:18.700779 2203 raft_consensus.cc:3037] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.700881 1944 raft_consensus.cc:3037] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.702980 2203 raft_consensus.cc:2454] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:18.703495 1944 raft_consensus.cc:2454] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:18.703596 2008 leader_election.cc:302] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:18.703994 3763 raft_consensus.cc:2781] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:18.704180 3763 raft_consensus.cc:686] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:18.704469 3763 consensus_queue.cc:237] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:18.708292 1668 catalog_manager.cc:5484] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:18.714972 3763 raft_consensus.cc:482] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:18.715179 3763 raft_consensus.cc:504] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:18.715890 3763 leader_election.cc:288] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:18.716383 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d9b1b3c23241d49bfcb914a5bd1777" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:18.716411 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d9b1b3c23241d49bfcb914a5bd1777" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:18.716634 2203 raft_consensus.cc:2454] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:18.716603 1944 raft_consensus.cc:2454] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:18.717026 2009 leader_election.cc:302] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:18.717394 3763 raft_consensus.cc:2781] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:18.717528 3763 raft_consensus.cc:482] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.717660 3763 raft_consensus.cc:3037] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.719801 3763 raft_consensus.cc:504] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:18.720474 3763 leader_election.cc:288] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:18.721220 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d9b1b3c23241d49bfcb914a5bd1777" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:18.721477 2203 raft_consensus.cc:3037] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.722117 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89d9b1b3c23241d49bfcb914a5bd1777" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:18.722363 1944 raft_consensus.cc:3037] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.724334 2203 raft_consensus.cc:2454] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:18.724841 2008 leader_election.cc:302] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:18.725221 1944 raft_consensus.cc:2454] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:18.725231 3763 raft_consensus.cc:2781] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:18.725383 3763 raft_consensus.cc:686] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:18.725903 3763 consensus_queue.cc:237] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:18.734230 1668 catalog_manager.cc:5484] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:18.805459 3889 raft_consensus.cc:482] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:18.805688 3889 raft_consensus.cc:504] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:18.806509 3889 leader_election.cc:288] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:18.807089 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa111e798e3f46d2a5bc1eebfe28f8bf" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:18.807128 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa111e798e3f46d2a5bc1eebfe28f8bf" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:18.807391 2203 raft_consensus.cc:2454] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:18.807399 1813 raft_consensus.cc:2454] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:18.807899 1878 leader_election.cc:302] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:18.808211 3889 raft_consensus.cc:2781] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:18.808315 3889 raft_consensus.cc:482] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.808382 3889 raft_consensus.cc:3037] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.810549 3889 raft_consensus.cc:504] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:18.811800 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa111e798e3f46d2a5bc1eebfe28f8bf" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:18.811888 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa111e798e3f46d2a5bc1eebfe28f8bf" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:18.812042 2203 raft_consensus.cc:3037] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.812116 1813 raft_consensus.cc:3037] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:18.815037 1813 raft_consensus.cc:2454] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:18.815055 2203 raft_consensus.cc:2454] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:18.815357 3889 leader_election.cc:288] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:18.815620 1879 leader_election.cc:302] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:18.815970 3889 raft_consensus.cc:2781] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:18.816115 3889 raft_consensus.cc:686] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:18.816375 3889 consensus_queue.cc:237] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:18.819866 1669 catalog_manager.cc:5484] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:19.155274 3763 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:19.201818 3763 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:19.225144 3929 consensus_queue.cc:1035] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:19.232067 3848 consensus_queue.cc:1035] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:19.278764 3889 consensus_queue.cc:1035] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:19.281702 3924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:19.282181 3924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:12:19.290313 3889 consensus_queue.cc:1035] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:19.296609 3924 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
INSERT report
rows total: 2000
time total: 286.917 ms
time per row: 0.143458 ms
I20240503 08:12:19.752007 1669 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:19.762094 1793 tablet_service.cc:1467] Processing CreateTablet for tablet f1cc728faa604b7cb9e250961e33f2df (DEFAULT_TABLE table=rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.762234 2054 tablet_service.cc:1467] Processing CreateTablet for tablet f1cc728faa604b7cb9e250961e33f2df (DEFAULT_TABLE table=rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.762639 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cc728faa604b7cb9e250961e33f2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.762689 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cc728faa604b7cb9e250961e33f2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.762746 1924 tablet_service.cc:1467] Processing CreateTablet for tablet f1cc728faa604b7cb9e250961e33f2df (DEFAULT_TABLE table=rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.763183 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cc728faa604b7cb9e250961e33f2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.765095 1792 tablet_service.cc:1467] Processing CreateTablet for tablet f5df898b258f40d882ecba23b34e1c34 (DEFAULT_TABLE table=rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.765156 1791 tablet_service.cc:1467] Processing CreateTablet for tablet aa19fc0ea2104f17aa69583702203c3c (DEFAULT_TABLE table=rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.765565 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5df898b258f40d882ecba23b34e1c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.765729 2184 tablet_service.cc:1467] Processing CreateTablet for tablet f5df898b258f40d882ecba23b34e1c34 (DEFAULT_TABLE table=rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.766144 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5df898b258f40d882ecba23b34e1c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.767769 1923 tablet_service.cc:1467] Processing CreateTablet for tablet aa19fc0ea2104f17aa69583702203c3c (DEFAULT_TABLE table=rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.768186 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa19fc0ea2104f17aa69583702203c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.773216 2052 tablet_service.cc:1467] Processing CreateTablet for tablet aa19fc0ea2104f17aa69583702203c3c (DEFAULT_TABLE table=rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.773294 2053 tablet_service.cc:1467] Processing CreateTablet for tablet f5df898b258f40d882ecba23b34e1c34 (DEFAULT_TABLE table=rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:19.773674 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa19fc0ea2104f17aa69583702203c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.775352 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5df898b258f40d882ecba23b34e1c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.788621 3993 tablet_bootstrap.cc:492] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:19.790132 3993 tablet_bootstrap.cc:654] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.798202 3995 tablet_bootstrap.cc:492] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:19.800485 3995 tablet_bootstrap.cc:654] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.802445 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa19fc0ea2104f17aa69583702203c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:19.810420 3997 tablet_bootstrap.cc:492] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:19.811805 3995 tablet_bootstrap.cc:492] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:19.812281 3995 ts_tablet_manager.cc:1367] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20240503 08:12:19.812541 3997 tablet_bootstrap.cc:654] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.811764 3992 tablet_bootstrap.cc:492] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:19.813223 3995 raft_consensus.cc:348] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.813480 3995 raft_consensus.cc:374] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.813639 3995 raft_consensus.cc:729] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.814617 3993 tablet_bootstrap.cc:492] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:19.814873 3993 ts_tablet_manager.cc:1367] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.016s
I20240503 08:12:19.815825 3993 raft_consensus.cc:348] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:19.816031 3992 tablet_bootstrap.cc:654] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.815783 3995 consensus_queue.cc:260] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.816174 3993 raft_consensus.cc:374] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.816319 3993 raft_consensus.cc:729] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.816684 3993 consensus_queue.cc:260] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:19.819064 3993 ts_tablet_manager.cc:1398] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:19.820634 3992 tablet_bootstrap.cc:492] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:19.820632 3995 ts_tablet_manager.cc:1398] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20240503 08:12:19.820797 3992 ts_tablet_manager.cc:1367] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20240503 08:12:19.821084 3995 tablet_bootstrap.cc:492] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:19.821614 3992 raft_consensus.cc:348] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.821913 3992 raft_consensus.cc:374] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.822077 3992 raft_consensus.cc:729] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.822547 3992 consensus_queue.cc:260] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.823293 3992 ts_tablet_manager.cc:1398] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:12:19.823494 3995 tablet_bootstrap.cc:654] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.823868 3992 tablet_bootstrap.cc:492] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:19.825533 3992 tablet_bootstrap.cc:654] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.834710 3997 tablet_bootstrap.cc:492] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:19.835180 3992 tablet_bootstrap.cc:492] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:19.835222 3997 ts_tablet_manager.cc:1367] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.001s
I20240503 08:12:19.835397 3992 ts_tablet_manager.cc:1367] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240503 08:12:19.836427 3997 raft_consensus.cc:348] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.836505 3992 raft_consensus.cc:348] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.836748 3992 raft_consensus.cc:374] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.836786 3997 raft_consensus.cc:374] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.836841 3992 raft_consensus.cc:729] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.836987 3997 raft_consensus.cc:729] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.837127 3992 consensus_queue.cc:260] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.837410 3997 consensus_queue.cc:260] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.837754 3992 ts_tablet_manager.cc:1398] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:19.838152 3992 tablet_bootstrap.cc:492] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:19.838423 3997 ts_tablet_manager.cc:1398] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:19.838896 3997 tablet_bootstrap.cc:492] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:19.839887 3992 tablet_bootstrap.cc:654] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.840926 3997 tablet_bootstrap.cc:654] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.843266 3995 tablet_bootstrap.cc:492] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:19.843549 3995 ts_tablet_manager.cc:1367] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.006s
I20240503 08:12:19.844655 3995 raft_consensus.cc:348] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.846848 3995 raft_consensus.cc:374] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.846940 3995 raft_consensus.cc:729] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.846978 3997 tablet_bootstrap.cc:492] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:19.847298 3997 ts_tablet_manager.cc:1367] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240503 08:12:19.847242 3995 consensus_queue.cc:260] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.847959 3992 tablet_bootstrap.cc:492] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:19.848129 3992 ts_tablet_manager.cc:1367] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240503 08:12:19.848268 3997 raft_consensus.cc:348] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:19.848496 3997 raft_consensus.cc:374] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.848584 3997 raft_consensus.cc:729] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.848850 3997 consensus_queue.cc:260] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:19.849047 3995 ts_tablet_manager.cc:1398] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 08:12:19.849403 3997 ts_tablet_manager.cc:1398] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:19.849728 3997 tablet_bootstrap.cc:492] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:19.850900 3992 raft_consensus.cc:348] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:19.851136 3992 raft_consensus.cc:374] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.851235 3992 raft_consensus.cc:729] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.851589 3997 tablet_bootstrap.cc:654] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:19.851495 3992 consensus_queue.cc:260] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:19.852082 3992 ts_tablet_manager.cc:1398] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:19.860805 3997 tablet_bootstrap.cc:492] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:19.861006 3997 ts_tablet_manager.cc:1367] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240503 08:12:19.861963 3997 raft_consensus.cc:348] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.862223 3997 raft_consensus.cc:374] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:19.862360 3997 raft_consensus.cc:729] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.862651 3997 consensus_queue.cc:260] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.863451 3997 ts_tablet_manager.cc:1398] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:19.891144 3842 raft_consensus.cc:482] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:19.891395 3842 raft_consensus.cc:504] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:19.892494 3842 leader_election.cc:288] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:19.893133 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5df898b258f40d882ecba23b34e1c34" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:19.893241 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5df898b258f40d882ecba23b34e1c34" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:19.893443 1813 raft_consensus.cc:2454] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:19.893478 2073 raft_consensus.cc:2454] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:19.893903 2139 leader_election.cc:302] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:19.894323 4005 raft_consensus.cc:2781] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:19.894433 4005 raft_consensus.cc:482] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:19.894564 4005 raft_consensus.cc:3037] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:19.896781 4005 raft_consensus.cc:504] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:19.897477 4005 leader_election.cc:288] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:19.897938 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5df898b258f40d882ecba23b34e1c34" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:19.898061 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5df898b258f40d882ecba23b34e1c34" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:19.898162 1813 raft_consensus.cc:3037] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:19.898272 2072 raft_consensus.cc:3037] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:19.901230 1813 raft_consensus.cc:2454] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:19.901235 2072 raft_consensus.cc:2454] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:19.901794 2139 leader_election.cc:302] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:19.902143 4005 raft_consensus.cc:2781] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:19.902335 4005 raft_consensus.cc:686] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:19.902629 4005 consensus_queue.cc:237] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:19.906343 1669 catalog_manager.cc:5484] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:19.963112 3976 raft_consensus.cc:482] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:19.963404 3976 raft_consensus.cc:504] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.964072 3976 leader_election.cc:288] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:19.964681 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cc728faa604b7cb9e250961e33f2df" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:19.964705 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cc728faa604b7cb9e250961e33f2df" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:19.964951 1944 raft_consensus.cc:2454] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:19.964970 1813 raft_consensus.cc:2454] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:19.965458 2009 leader_election.cc:302] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:19.965766 3976 raft_consensus.cc:2781] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:19.965885 3976 raft_consensus.cc:482] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:19.965973 3976 raft_consensus.cc:3037] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:19.967916 3976 raft_consensus.cc:504] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.968547 3976 leader_election.cc:288] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:19.969112 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cc728faa604b7cb9e250961e33f2df" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:19.969175 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cc728faa604b7cb9e250961e33f2df" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:19.969377 1813 raft_consensus.cc:3037] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:19.969388 1944 raft_consensus.cc:3037] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:19.972364 1944 raft_consensus.cc:2454] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:19.972347 1813 raft_consensus.cc:2454] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:19.972888 2009 leader_election.cc:302] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:19.973198 3976 raft_consensus.cc:2781] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:19.973368 3976 raft_consensus.cc:686] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:19.973654 3976 consensus_queue.cc:237] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.975134 3976 raft_consensus.cc:482] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:19.975359 3976 raft_consensus.cc:504] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.976066 3976 leader_election.cc:288] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:19.977162 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa19fc0ea2104f17aa69583702203c3c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:19.977146 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa19fc0ea2104f17aa69583702203c3c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:19.977383 1813 raft_consensus.cc:2454] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:19.977396 1944 raft_consensus.cc:2454] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:19.977403 1669 catalog_manager.cc:5484] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:19.977917 2009 leader_election.cc:302] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:19.978230 3976 raft_consensus.cc:2781] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:19.978341 3976 raft_consensus.cc:482] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:19.978463 3976 raft_consensus.cc:3037] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:19.981467 3976 raft_consensus.cc:504] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.982157 3976 leader_election.cc:288] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:19.982573 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa19fc0ea2104f17aa69583702203c3c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:19.982834 1813 raft_consensus.cc:3037] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:19.985330 1813 raft_consensus.cc:2454] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:19.985826 2009 leader_election.cc:302] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:19.986132 3976 raft_consensus.cc:2781] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:19.986088 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa19fc0ea2104f17aa69583702203c3c" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:19.986336 3976 raft_consensus.cc:686] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:19.986358 1944 raft_consensus.cc:3037] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:19.986625 3976 consensus_queue.cc:237] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:19.988993 1944 raft_consensus.cc:2454] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:19.993494 1665 catalog_manager.cc:5484] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:20.328668 4005 consensus_queue.cc:1035] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:20.337052 4005 consensus_queue.cc:1035] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:20.395401 3763 consensus_queue.cc:1035] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:20.406826 3763 consensus_queue.cc:1035] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:20.434244 4009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:20.434641 4009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:20.442947 4009 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:20.471200 3978 consensus_queue.cc:1035] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:20.476725 3978 consensus_queue.cc:1035] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_14'
INSERT report
rows total: 2000
time total: 305.404 ms
time per row: 0.152702 ms
I20240503 08:12:20.915097 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:20.925930 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 53197deeb8d7439c90b019a6a90857e0 (DEFAULT_TABLE table=rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.926218 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 53197deeb8d7439c90b019a6a90857e0 (DEFAULT_TABLE table=rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.926482 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53197deeb8d7439c90b019a6a90857e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.926693 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53197deeb8d7439c90b019a6a90857e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.927062 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 53197deeb8d7439c90b019a6a90857e0 (DEFAULT_TABLE table=rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.927544 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53197deeb8d7439c90b019a6a90857e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.927692 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 1811bd652d2c4afb9f52bcd64ceb2acb (DEFAULT_TABLE table=rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.928133 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1811bd652d2c4afb9f52bcd64ceb2acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.928252 2182 tablet_service.cc:1467] Processing CreateTablet for tablet 1811bd652d2c4afb9f52bcd64ceb2acb (DEFAULT_TABLE table=rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.928640 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1811bd652d2c4afb9f52bcd64ceb2acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.926477 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 1811bd652d2c4afb9f52bcd64ceb2acb (DEFAULT_TABLE table=rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.932200 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1811bd652d2c4afb9f52bcd64ceb2acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.936858 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 358cc7bd631d432385d6205db86cbfd4 (DEFAULT_TABLE table=rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.937044 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 358cc7bd631d432385d6205db86cbfd4 (DEFAULT_TABLE table=rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.937330 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358cc7bd631d432385d6205db86cbfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.937585 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358cc7bd631d432385d6205db86cbfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.944496 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 358cc7bd631d432385d6205db86cbfd4 (DEFAULT_TABLE table=rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:20.945165 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358cc7bd631d432385d6205db86cbfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:20.945312 4076 tablet_bootstrap.cc:492] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:20.945555 4077 tablet_bootstrap.cc:492] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:20.947299 4076 tablet_bootstrap.cc:654] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:20.947664 4077 tablet_bootstrap.cc:654] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:20.955227 4076 tablet_bootstrap.cc:492] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:20.955457 4076 ts_tablet_manager.cc:1367] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240503 08:12:20.956324 4076 raft_consensus.cc:348] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:20.956728 4076 raft_consensus.cc:374] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.956851 4076 raft_consensus.cc:729] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.957113 4076 consensus_queue.cc:260] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:20.957993 4077 tablet_bootstrap.cc:492] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:20.958230 4077 ts_tablet_manager.cc:1367] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240503 08:12:20.958988 4076 ts_tablet_manager.cc:1398] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:20.959213 4077 raft_consensus.cc:348] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:20.959491 4077 raft_consensus.cc:374] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.959802 4077 raft_consensus.cc:729] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.959926 4076 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:20.960086 4077 consensus_queue.cc:260] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:20.960912 4077 ts_tablet_manager.cc:1398] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:20.961421 4077 tablet_bootstrap.cc:492] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:20.962026 4076 tablet_bootstrap.cc:654] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:20.962913 4075 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:20.963418 4077 tablet_bootstrap.cc:654] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:20.965757 4075 tablet_bootstrap.cc:654] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:20.971144 4076 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:20.971385 4076 ts_tablet_manager.cc:1367] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240503 08:12:20.972283 4076 raft_consensus.cc:348] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:20.972537 4076 raft_consensus.cc:374] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.972635 4076 raft_consensus.cc:729] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.972952 4076 consensus_queue.cc:260] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:20.973104 4077 tablet_bootstrap.cc:492] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:20.973294 4077 ts_tablet_manager.cc:1367] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240503 08:12:20.973646 4076 ts_tablet_manager.cc:1398] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:20.974014 4075 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:20.974179 4075 ts_tablet_manager.cc:1367] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 08:12:20.974749 4077 raft_consensus.cc:348] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:20.975009 4077 raft_consensus.cc:374] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.975103 4077 raft_consensus.cc:729] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.975065 4075 raft_consensus.cc:348] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:20.975296 4075 raft_consensus.cc:374] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.975412 4075 raft_consensus.cc:729] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.975410 4077 consensus_queue.cc:260] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:20.976063 4077 ts_tablet_manager.cc:1398] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:20.979287 4079 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:20.981134 4079 tablet_bootstrap.cc:654] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:20.981333 4075 consensus_queue.cc:260] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:20.982084 4075 ts_tablet_manager.cc:1398] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:12:20.982511 4075 tablet_bootstrap.cc:492] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:20.983451 4079 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:20.983672 4079 ts_tablet_manager.cc:1367] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 08:12:20.984333 4075 tablet_bootstrap.cc:654] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:20.984750 4079 raft_consensus.cc:348] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:20.985028 4079 raft_consensus.cc:374] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.985157 4079 raft_consensus.cc:729] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.985411 4079 consensus_queue.cc:260] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:20.986889 4075 tablet_bootstrap.cc:492] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:20.987114 4075 ts_tablet_manager.cc:1367] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:12:20.987354 4079 ts_tablet_manager.cc:1398] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:20.987689 4079 tablet_bootstrap.cc:492] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:20.988085 4075 raft_consensus.cc:348] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:20.988349 4075 raft_consensus.cc:374] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:20.988449 4075 raft_consensus.cc:729] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:20.988746 4075 consensus_queue.cc:260] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:20.989274 4075 ts_tablet_manager.cc:1398] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:20.989638 4075 tablet_bootstrap.cc:492] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:20.990026 4079 tablet_bootstrap.cc:654] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:20.991601 4075 tablet_bootstrap.cc:654] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:20.998507 4079 tablet_bootstrap.cc:492] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:20.998745 4079 ts_tablet_manager.cc:1367] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240503 08:12:20.999770 4079 raft_consensus.cc:348] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:21.000025 4079 raft_consensus.cc:374] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:21.000119 4079 raft_consensus.cc:729] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.000360 4079 consensus_queue.cc:260] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:21.000985 4079 ts_tablet_manager.cc:1398] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:21.002461 4075 tablet_bootstrap.cc:492] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:21.002646 4075 ts_tablet_manager.cc:1367] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240503 08:12:21.003605 4075 raft_consensus.cc:348] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:21.003824 4075 raft_consensus.cc:374] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:21.003921 4075 raft_consensus.cc:729] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.004158 4075 consensus_queue.cc:260] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:21.004725 4075 ts_tablet_manager.cc:1398] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:21.071373 4018 raft_consensus.cc:482] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:21.071575 4018 raft_consensus.cc:504] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:21.072279 4018 leader_election.cc:288] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:21.072829 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53197deeb8d7439c90b019a6a90857e0" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:21.072834 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53197deeb8d7439c90b019a6a90857e0" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:21.073066 2203 raft_consensus.cc:2454] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:21.073076 1813 raft_consensus.cc:2454] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:21.073654 1878 leader_election.cc:302] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:21.073953 4018 raft_consensus.cc:2781] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:21.074088 4018 raft_consensus.cc:482] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:21.074195 4018 raft_consensus.cc:3037] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.076292 4018 raft_consensus.cc:504] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:21.076982 4018 leader_election.cc:288] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:21.077523 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53197deeb8d7439c90b019a6a90857e0" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:21.077541 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53197deeb8d7439c90b019a6a90857e0" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:21.077755 1813 raft_consensus.cc:3037] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.077773 2203 raft_consensus.cc:3037] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.079978 2203 raft_consensus.cc:2454] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:21.079964 1813 raft_consensus.cc:2454] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:21.080437 1878 leader_election.cc:302] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:21.080919 4018 raft_consensus.cc:2781] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:21.081086 4018 raft_consensus.cc:686] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:21.081368 3976 raft_consensus.cc:482] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:21.081409 4018 consensus_queue.cc:237] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:21.081549 3976 raft_consensus.cc:504] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:21.082336 3976 leader_election.cc:288] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:21.082839 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811bd652d2c4afb9f52bcd64ceb2acb" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:21.083079 1813 raft_consensus.cc:2454] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:21.083479 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811bd652d2c4afb9f52bcd64ceb2acb" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:21.083683 2203 raft_consensus.cc:2454] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:21.083638 2009 leader_election.cc:302] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:21.084165 3978 raft_consensus.cc:2781] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:21.084280 3978 raft_consensus.cc:482] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:21.084380 3978 raft_consensus.cc:3037] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.086997 3978 raft_consensus.cc:504] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:21.087790 3978 leader_election.cc:288] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:21.088358 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811bd652d2c4afb9f52bcd64ceb2acb" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:21.088343 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811bd652d2c4afb9f52bcd64ceb2acb" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:21.088681 1813 raft_consensus.cc:3037] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.088702 2203 raft_consensus.cc:3037] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.090255 1668 catalog_manager.cc:5484] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:21.090893 2203 raft_consensus.cc:2454] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:21.091429 2008 leader_election.cc:302] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:21.091621 1813 raft_consensus.cc:2454] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:21.091784 3978 raft_consensus.cc:2781] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:21.091957 3978 raft_consensus.cc:686] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:21.092236 3978 consensus_queue.cc:237] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:21.095908 1668 catalog_manager.cc:5484] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:21.299926 3978 raft_consensus.cc:482] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:21.300140 3978 raft_consensus.cc:504] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:21.300849 3978 leader_election.cc:288] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:21.301321 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358cc7bd631d432385d6205db86cbfd4" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:21.301484 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358cc7bd631d432385d6205db86cbfd4" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:21.301609 2202 raft_consensus.cc:2454] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:21.301770 1942 raft_consensus.cc:2454] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:21.302197 2008 leader_election.cc:302] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:21.302518 3978 raft_consensus.cc:2781] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:21.302630 3978 raft_consensus.cc:482] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:21.302726 3978 raft_consensus.cc:3037] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.304796 3978 raft_consensus.cc:504] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:21.305466 3978 leader_election.cc:288] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:21.306000 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358cc7bd631d432385d6205db86cbfd4" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:21.306041 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358cc7bd631d432385d6205db86cbfd4" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:21.306239 2202 raft_consensus.cc:3037] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.306270 1942 raft_consensus.cc:3037] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:21.308341 1942 raft_consensus.cc:2454] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:21.308917 2202 raft_consensus.cc:2454] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:21.309016 2009 leader_election.cc:302] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:21.309595 3978 raft_consensus.cc:2781] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:21.309765 3978 raft_consensus.cc:686] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:21.310060 3978 consensus_queue.cc:237] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:21.313124 1668 catalog_manager.cc:5484] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:21.563421 4018 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:21.586120 4018 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:21.607942 3978 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:21.613950 3763 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:21.719098 4091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:21.719637 4091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:12:21.730010 3978 consensus_queue.cc:1035] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:21.732235 4091 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:21.738337 3976 consensus_queue.cc:1035] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_15'
W20240503 08:12:22.025012 2008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.196:42183, user_credentials={real_user=slave}} blocked reactor thread for 78653.1us
W20240503 08:12:22.025364 2009 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.194:41975, user_credentials={real_user=slave}} blocked reactor thread for 75050us
INSERT report
rows total: 2000
time total: 305.645 ms
time per row: 0.152823 ms
I20240503 08:12:22.221163 1669 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:22.231629 1791 tablet_service.cc:1467] Processing CreateTablet for tablet dc82c5ccee9e47e4bde9ff51860fd8a0 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.232190 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc82c5ccee9e47e4bde9ff51860fd8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.232255 1793 tablet_service.cc:1467] Processing CreateTablet for tablet fabf21f75e4944bebdf8da5581c4af34 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.232703 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabf21f75e4944bebdf8da5581c4af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.232609 2054 tablet_service.cc:1467] Processing CreateTablet for tablet fabf21f75e4944bebdf8da5581c4af34 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.233068 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabf21f75e4944bebdf8da5581c4af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.236877 2182 tablet_service.cc:1467] Processing CreateTablet for tablet dc82c5ccee9e47e4bde9ff51860fd8a0 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.237149 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 8d45993944a946adacf5dd05ec67e3b7 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.237377 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc82c5ccee9e47e4bde9ff51860fd8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.237649 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d45993944a946adacf5dd05ec67e3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.239261 1924 tablet_service.cc:1467] Processing CreateTablet for tablet fabf21f75e4944bebdf8da5581c4af34 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.239727 1923 tablet_service.cc:1467] Processing CreateTablet for tablet dc82c5ccee9e47e4bde9ff51860fd8a0 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.239750 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabf21f75e4944bebdf8da5581c4af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.243566 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc82c5ccee9e47e4bde9ff51860fd8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.248880 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 8d45993944a946adacf5dd05ec67e3b7 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.249395 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d45993944a946adacf5dd05ec67e3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.251523 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 8d45993944a946adacf5dd05ec67e3b7 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.251979 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d45993944a946adacf5dd05ec67e3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.259241 4156 tablet_bootstrap.cc:492] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:22.261101 4157 tablet_bootstrap.cc:492] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:22.261231 4156 tablet_bootstrap.cc:654] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.265488 4156 tablet_bootstrap.cc:492] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:22.265969 4156 ts_tablet_manager.cc:1367] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 08:12:22.266961 4156 raft_consensus.cc:348] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:22.267374 4156 raft_consensus.cc:374] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.267531 4156 raft_consensus.cc:729] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.267880 4156 consensus_queue.cc:260] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:22.268823 4156 ts_tablet_manager.cc:1398] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:22.269306 4156 tablet_bootstrap.cc:492] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:22.270872 4158 tablet_bootstrap.cc:492] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:22.271463 4156 tablet_bootstrap.cc:654] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.272001 4157 tablet_bootstrap.cc:654] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.273649 4158 tablet_bootstrap.cc:654] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.283710 4160 tablet_bootstrap.cc:492] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:22.283957 4158 tablet_bootstrap.cc:492] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:22.285111 4158 ts_tablet_manager.cc:1367] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20240503 08:12:22.285794 4156 tablet_bootstrap.cc:492] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:22.286051 4156 ts_tablet_manager.cc:1367] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20240503 08:12:22.286170 4160 tablet_bootstrap.cc:654] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.286178 4158 raft_consensus.cc:348] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:22.286437 4158 raft_consensus.cc:374] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.286588 4158 raft_consensus.cc:729] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.286989 4156 raft_consensus.cc:348] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:22.287082 4158 consensus_queue.cc:260] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:22.287242 4156 raft_consensus.cc:374] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.287369 4156 raft_consensus.cc:729] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.287703 4156 consensus_queue.cc:260] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:22.288388 4156 ts_tablet_manager.cc:1398] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:22.288849 4156 tablet_bootstrap.cc:492] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:22.289474 4158 ts_tablet_manager.cc:1398] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:22.289808 4158 tablet_bootstrap.cc:492] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:22.290879 4156 tablet_bootstrap.cc:654] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.291517 4158 tablet_bootstrap.cc:654] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.292768 4157 tablet_bootstrap.cc:492] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:22.292994 4157 ts_tablet_manager.cc:1367] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.008s
I20240503 08:12:22.296541 4157 raft_consensus.cc:348] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:22.296806 4157 raft_consensus.cc:374] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.297048 4157 raft_consensus.cc:729] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.297430 4157 consensus_queue.cc:260] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:22.302582 4158 tablet_bootstrap.cc:492] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:22.302809 4158 ts_tablet_manager.cc:1367] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20240503 08:12:22.304008 4158 raft_consensus.cc:348] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:22.304366 4158 raft_consensus.cc:374] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.304426 4157 ts_tablet_manager.cc:1398] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20240503 08:12:22.304520 4158 raft_consensus.cc:729] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.304842 4158 consensus_queue.cc:260] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:22.305150 4157 tablet_bootstrap.cc:492] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:22.306583 4158 ts_tablet_manager.cc:1398] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:22.310345 4157 tablet_bootstrap.cc:654] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.315073 4160 tablet_bootstrap.cc:492] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:22.315387 4160 ts_tablet_manager.cc:1367] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.003s
I20240503 08:12:22.316301 4160 raft_consensus.cc:348] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:22.316529 4160 raft_consensus.cc:374] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.316646 4160 raft_consensus.cc:729] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.316931 4160 consensus_queue.cc:260] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:22.318518 4156 tablet_bootstrap.cc:492] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:22.318768 4156 ts_tablet_manager.cc:1367] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.004s
I20240503 08:12:22.318931 4160 ts_tablet_manager.cc:1398] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:22.319275 4160 tablet_bootstrap.cc:492] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:22.320335 4156 raft_consensus.cc:348] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:22.320674 4156 raft_consensus.cc:374] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.320835 4156 raft_consensus.cc:729] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.321156 4156 consensus_queue.cc:260] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:22.321964 4156 ts_tablet_manager.cc:1398] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:22.322613 4160 tablet_bootstrap.cc:654] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.323486 4157 tablet_bootstrap.cc:492] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:22.323679 4157 ts_tablet_manager.cc:1367] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.002s
I20240503 08:12:22.324535 4157 raft_consensus.cc:348] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:22.324800 4157 raft_consensus.cc:374] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.324913 4157 raft_consensus.cc:729] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.325266 4157 consensus_queue.cc:260] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:22.326560 4157 ts_tablet_manager.cc:1398] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:22.328114 3925 raft_consensus.cc:482] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:22.328431 3925 raft_consensus.cc:504] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:22.329917 3925 leader_election.cc:288] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:22.331641 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabf21f75e4944bebdf8da5581c4af34" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:22.332072 2073 raft_consensus.cc:2454] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:22.332976 1747 leader_election.cc:302] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:22.333854 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabf21f75e4944bebdf8da5581c4af34" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:22.334090 1944 raft_consensus.cc:2454] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:22.335296 4092 raft_consensus.cc:2781] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:22.335481 4092 raft_consensus.cc:482] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.335654 4092 raft_consensus.cc:3037] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.339781 4092 raft_consensus.cc:504] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:22.341197 4092 leader_election.cc:288] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:22.341797 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabf21f75e4944bebdf8da5581c4af34" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:22.342051 1944 raft_consensus.cc:3037] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.342237 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabf21f75e4944bebdf8da5581c4af34" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:22.342557 2073 raft_consensus.cc:3037] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.344832 1944 raft_consensus.cc:2454] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:22.345695 1748 leader_election.cc:302] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:22.345995 4092 raft_consensus.cc:2781] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:22.346228 4092 raft_consensus.cc:686] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:22.346554 4092 consensus_queue.cc:237] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:22.348263 2073 raft_consensus.cc:2454] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:22.349236 4160 tablet_bootstrap.cc:492] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:22.349442 4160 ts_tablet_manager.cc:1367] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.017s
I20240503 08:12:22.350376 4160 raft_consensus.cc:348] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:22.350625 4160 raft_consensus.cc:374] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.350718 4160 raft_consensus.cc:729] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.350986 4160 consensus_queue.cc:260] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:22.352638 4160 ts_tablet_manager.cc:1398] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:12:22.353189 1669 catalog_manager.cc:5484] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:22.483515 4057 raft_consensus.cc:482] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:22.483765 4057 raft_consensus.cc:504] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:22.484766 4057 leader_election.cc:288] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:22.485240 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c5ccee9e47e4bde9ff51860fd8a0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:22.485378 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c5ccee9e47e4bde9ff51860fd8a0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:22.485515 1811 raft_consensus.cc:2454] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:22.485677 1944 raft_consensus.cc:2454] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:22.486011 2139 leader_election.cc:302] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:22.486267 4057 raft_consensus.cc:2781] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:22.486392 4057 raft_consensus.cc:482] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.486485 4057 raft_consensus.cc:3037] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.488512 4057 raft_consensus.cc:504] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:22.489141 4057 leader_election.cc:288] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:22.489722 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c5ccee9e47e4bde9ff51860fd8a0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:22.489765 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c5ccee9e47e4bde9ff51860fd8a0" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:22.489984 1811 raft_consensus.cc:3037] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.490034 1944 raft_consensus.cc:3037] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.492185 1811 raft_consensus.cc:2454] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:22.492288 1944 raft_consensus.cc:2454] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:22.492738 2139 leader_election.cc:302] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:22.493098 4057 raft_consensus.cc:2781] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:22.493263 4057 raft_consensus.cc:686] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:22.493530 4057 consensus_queue.cc:237] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:22.497242 1669 catalog_manager.cc:5484] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:22.640920 4057 raft_consensus.cc:482] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:22.641134 4057 raft_consensus.cc:504] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:22.641844 4057 leader_election.cc:288] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:22.642401 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d45993944a946adacf5dd05ec67e3b7" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:22.642400 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d45993944a946adacf5dd05ec67e3b7" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:22.642642 2073 raft_consensus.cc:2454] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:22.642650 1811 raft_consensus.cc:2454] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:22.643074 2139 leader_election.cc:302] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:22.643415 4057 raft_consensus.cc:2781] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:22.643508 4057 raft_consensus.cc:482] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.643674 4057 raft_consensus.cc:3037] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.645609 4057 raft_consensus.cc:504] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:22.646214 4057 leader_election.cc:288] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:22.646744 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d45993944a946adacf5dd05ec67e3b7" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:22.646739 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d45993944a946adacf5dd05ec67e3b7" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:22.646926 2073 raft_consensus.cc:3037] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.646930 1811 raft_consensus.cc:3037] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.650086 2073 raft_consensus.cc:2454] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:22.650995 2138 leader_election.cc:302] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:22.654186 4169 raft_consensus.cc:2781] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:22.654440 1811 raft_consensus.cc:2454] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:22.654677 4169 raft_consensus.cc:686] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:22.655047 4169 consensus_queue.cc:237] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:22.662494 1669 catalog_manager.cc:5484] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:22.734602 4092 consensus_queue.cc:1035] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:22.755712 3925 consensus_queue.cc:1035] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:22.939652 4057 consensus_queue.cc:1035] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:22.949842 4181 consensus_queue.cc:1035] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:23.067008 4181 consensus_queue.cc:1035] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:23.074374 4169 consensus_queue.cc:1035] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:23.124416 4173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:23.124747 4173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:23.133708 4173 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
INSERT report
rows total: 2000
time total: 238.787 ms
time per row: 0.119394 ms
I20240503 08:12:23.543941 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:23.554071 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 674e558bf5db4f169f8cdcb08c40f5fb (DEFAULT_TABLE table=rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.554610 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674e558bf5db4f169f8cdcb08c40f5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.554605 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 674e558bf5db4f169f8cdcb08c40f5fb (DEFAULT_TABLE table=rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.554728 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 133911d44b394e24a89a5d9443230d32 (DEFAULT_TABLE table=rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.555032 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674e558bf5db4f169f8cdcb08c40f5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.555153 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133911d44b394e24a89a5d9443230d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.555821 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 674e558bf5db4f169f8cdcb08c40f5fb (DEFAULT_TABLE table=rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.556254 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674e558bf5db4f169f8cdcb08c40f5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.562899 2182 tablet_service.cc:1467] Processing CreateTablet for tablet 133911d44b394e24a89a5d9443230d32 (DEFAULT_TABLE table=rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.563660 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133911d44b394e24a89a5d9443230d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.568075 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 133911d44b394e24a89a5d9443230d32 (DEFAULT_TABLE table=rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.568735 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133911d44b394e24a89a5d9443230d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.569331 4240 tablet_bootstrap.cc:492] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:23.571929 4240 tablet_bootstrap.cc:654] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.572350 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e (DEFAULT_TABLE table=rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.572809 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.573189 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e (DEFAULT_TABLE table=rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.573520 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.585666 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e (DEFAULT_TABLE table=rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:23.586213 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:23.600620 4244 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:23.603356 4244 tablet_bootstrap.cc:654] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.612869 4244 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:23.613694 4244 ts_tablet_manager.cc:1367] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20240503 08:12:23.615099 4244 raft_consensus.cc:348] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.617091 4244 raft_consensus.cc:374] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.617445 4244 raft_consensus.cc:729] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.618124 4244 consensus_queue.cc:260] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.619665 4244 ts_tablet_manager.cc:1398] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:23.620967 4244 tablet_bootstrap.cc:492] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:23.624634 4245 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:23.625613 4244 tablet_bootstrap.cc:654] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.628089 4241 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:23.628249 4245 tablet_bootstrap.cc:654] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.628883 4244 tablet_bootstrap.cc:492] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:23.629129 4244 ts_tablet_manager.cc:1367] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 08:12:23.630218 4244 raft_consensus.cc:348] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:23.630540 4244 raft_consensus.cc:374] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.630570 4241 tablet_bootstrap.cc:654] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.630666 4244 raft_consensus.cc:729] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.631047 4244 consensus_queue.cc:260] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:23.631834 4244 ts_tablet_manager.cc:1398] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:12:23.632356 4240 tablet_bootstrap.cc:492] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:23.632578 4240 ts_tablet_manager.cc:1367] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.038s
I20240503 08:12:23.632172 4244 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:23.633566 4240 raft_consensus.cc:348] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:23.634083 4240 raft_consensus.cc:374] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.634255 4240 raft_consensus.cc:729] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.634637 4244 tablet_bootstrap.cc:654] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.634586 4240 consensus_queue.cc:260] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:23.635628 4240 ts_tablet_manager.cc:1398] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:12:23.636132 4240 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:23.646006 4240 tablet_bootstrap.cc:654] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.657598 4245 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:23.657902 4245 ts_tablet_manager.cc:1367] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.004s
I20240503 08:12:23.659194 4245 raft_consensus.cc:348] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.659554 4245 raft_consensus.cc:374] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.659693 4245 raft_consensus.cc:729] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.660231 4245 consensus_queue.cc:260] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.664494 4245 ts_tablet_manager.cc:1398] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:23.667892 4241 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:23.667996 4243 raft_consensus.cc:482] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:23.668164 4241 ts_tablet_manager.cc:1367] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.040s user 0.017s sys 0.017s
I20240503 08:12:23.668206 4243 raft_consensus.cc:504] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.669067 4243 leader_election.cc:288] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:23.669447 4241 raft_consensus.cc:348] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.669831 4241 raft_consensus.cc:374] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.669991 4241 raft_consensus.cc:729] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.669922 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9b2eef42e4fff8cdf5f9dd7b8b31e" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:23.670179 1811 raft_consensus.cc:2454] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:23.670228 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9b2eef42e4fff8cdf5f9dd7b8b31e" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:23.670475 4241 consensus_queue.cc:260] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.671240 4241 ts_tablet_manager.cc:1398] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:23.671917 4241 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:23.671856 1879 leader_election.cc:302] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:23.672189 4243 raft_consensus.cc:2781] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:23.672318 4243 raft_consensus.cc:482] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:23.672422 4243 raft_consensus.cc:3037] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:12:23.672452 1878 leader_election.cc:341] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:23.674795 4241 tablet_bootstrap.cc:654] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.675349 4243 raft_consensus.cc:504] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.675627 4240 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:23.675810 4240 ts_tablet_manager.cc:1367] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.003s
I20240503 08:12:23.676757 4243 leader_election.cc:288] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:23.677136 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9b2eef42e4fff8cdf5f9dd7b8b31e" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:23.677441 1811 raft_consensus.cc:3037] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.677747 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9b2eef42e4fff8cdf5f9dd7b8b31e" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
W20240503 08:12:23.679447 1878 leader_election.cc:341] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:23.680272 1811 raft_consensus.cc:2454] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:23.680685 1879 leader_election.cc:302] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters: 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:23.680997 4243 raft_consensus.cc:2781] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:23.681213 4243 raft_consensus.cc:686] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:23.681245 4240 raft_consensus.cc:348] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.681497 4240 raft_consensus.cc:374] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.681579 4240 raft_consensus.cc:729] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.681749 4243 consensus_queue.cc:237] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.681831 4240 consensus_queue.cc:260] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.682567 4240 ts_tablet_manager.cc:1398] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:23.684888 4244 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:23.687109 4244 ts_tablet_manager.cc:1367] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.055s user 0.016s sys 0.018s
I20240503 08:12:23.688253 4244 raft_consensus.cc:348] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.689570 4244 raft_consensus.cc:374] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.689890 4244 raft_consensus.cc:729] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.690690 4244 consensus_queue.cc:260] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.691926 4244 ts_tablet_manager.cc:1398] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:23.696337 1669 catalog_manager.cc:5484] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:23.707087 4241 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:23.707289 4241 ts_tablet_manager.cc:1367] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.001s
I20240503 08:12:23.708005 4241 raft_consensus.cc:348] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.708186 4241 raft_consensus.cc:374] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.708266 4241 raft_consensus.cc:729] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.708516 4241 consensus_queue.cc:260] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.709041 4241 ts_tablet_manager.cc:1398] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:23.709444 4241 tablet_bootstrap.cc:492] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:23.710987 4241 tablet_bootstrap.cc:654] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:23.712989 4241 tablet_bootstrap.cc:492] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:23.713163 4241 ts_tablet_manager.cc:1367] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:23.713958 4241 raft_consensus.cc:348] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:23.714170 4241 raft_consensus.cc:374] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:23.714269 4241 raft_consensus.cc:729] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:23.714501 4241 consensus_queue.cc:260] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:23.715116 4241 ts_tablet_manager.cc:1398] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:23.724396 4134 raft_consensus.cc:482] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:23.724591 4134 raft_consensus.cc:504] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:23.725482 4134 leader_election.cc:288] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:23.725988 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133911d44b394e24a89a5d9443230d32" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:23.725998 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133911d44b394e24a89a5d9443230d32" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:23.726220 2202 raft_consensus.cc:2454] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:23.726234 1944 raft_consensus.cc:2454] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:23.726613 2008 leader_election.cc:302] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:23.726899 4134 raft_consensus.cc:2781] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:23.727020 4134 raft_consensus.cc:482] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:23.727109 4134 raft_consensus.cc:3037] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.729228 4134 raft_consensus.cc:504] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:23.729797 4134 leader_election.cc:288] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:23.730319 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133911d44b394e24a89a5d9443230d32" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:23.730340 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133911d44b394e24a89a5d9443230d32" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:23.730516 2202 raft_consensus.cc:3037] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.730556 1944 raft_consensus.cc:3037] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.732508 2202 raft_consensus.cc:2454] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:23.732995 2008 leader_election.cc:302] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:23.733337 4134 raft_consensus.cc:2781] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:23.733500 1944 raft_consensus.cc:2454] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:23.733508 4134 raft_consensus.cc:686] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:23.734226 4134 consensus_queue.cc:237] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:23.738008 1669 catalog_manager.cc:5484] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:23.790306 4243 raft_consensus.cc:482] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:23.790481 4243 raft_consensus.cc:504] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.791344 4243 leader_election.cc:288] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:23.791658 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674e558bf5db4f169f8cdcb08c40f5fb" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:23.791900 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674e558bf5db4f169f8cdcb08c40f5fb" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:23.791925 2072 raft_consensus.cc:2454] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:23.792143 2204 raft_consensus.cc:2454] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:23.792378 1878 leader_election.cc:302] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:23.792671 4243 raft_consensus.cc:2781] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:23.792752 4243 raft_consensus.cc:482] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:23.792863 4243 raft_consensus.cc:3037] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.794762 4243 raft_consensus.cc:504] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.795365 4243 leader_election.cc:288] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:23.795835 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674e558bf5db4f169f8cdcb08c40f5fb" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:23.795939 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674e558bf5db4f169f8cdcb08c40f5fb" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:23.796049 2072 raft_consensus.cc:3037] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.796190 2204 raft_consensus.cc:3037] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:23.798065 2072 raft_consensus.cc:2454] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:23.798130 2204 raft_consensus.cc:2454] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:23.798601 1878 leader_election.cc:302] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:23.798928 4243 raft_consensus.cc:2781] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:23.799089 4243 raft_consensus.cc:686] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:23.799378 4243 consensus_queue.cc:237] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:23.803134 1669 catalog_manager.cc:5484] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:24.069538 4057 raft_consensus.cc:482] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:24.069721 4057 raft_consensus.cc:504] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:24.070418 4057 leader_election.cc:288] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:24.070979 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9b2eef42e4fff8cdf5f9dd7b8b31e" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:24.071022 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9b2eef42e4fff8cdf5f9dd7b8b31e" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:24.071358 1813 raft_consensus.cc:2379] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 184c543cb05d49cc98ba1c56617be7a5 in current term 1: Already voted for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in this term.
I20240503 08:12:24.071817 2139 leader_election.cc:302] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e
I20240503 08:12:24.072115 4057 raft_consensus.cc:3037] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.074486 4057 raft_consensus.cc:2726] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:24.086926 4243 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:24.113071 4262 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:24.192713 4243 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:24.203508 4262 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:24.212255 4138 consensus_queue.cc:1035] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:24.223726 4257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:24.224313 4257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:24.240775 4257 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:24.240876 4134 consensus_queue.cc:1035] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_17'
INSERT report
rows total: 2000
time total: 237.767 ms
time per row: 0.118883 ms
I20240503 08:12:24.653093 1669 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:24.673756 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 668fe552878f42bbb5156c5e162b2630 (DEFAULT_TABLE table=rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.674283 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668fe552878f42bbb5156c5e162b2630. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.674275 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 668fe552878f42bbb5156c5e162b2630 (DEFAULT_TABLE table=rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.674722 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668fe552878f42bbb5156c5e162b2630. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.674685 1791 tablet_service.cc:1467] Processing CreateTablet for tablet dbbf914fa37043e8874583d1fdc914b6 (DEFAULT_TABLE table=rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.675132 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbf914fa37043e8874583d1fdc914b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.675758 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 668fe552878f42bbb5156c5e162b2630 (DEFAULT_TABLE table=rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.675745 2184 tablet_service.cc:1467] Processing CreateTablet for tablet dbbf914fa37043e8874583d1fdc914b6 (DEFAULT_TABLE table=rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.676172 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668fe552878f42bbb5156c5e162b2630. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.676179 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbf914fa37043e8874583d1fdc914b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.679859 2053 tablet_service.cc:1467] Processing CreateTablet for tablet dbbf914fa37043e8874583d1fdc914b6 (DEFAULT_TABLE table=rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.680305 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbf914fa37043e8874583d1fdc914b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.681584 1924 tablet_service.cc:1467] Processing CreateTablet for tablet c00305bb59f84487b2afcdc30910a663 (DEFAULT_TABLE table=rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.682044 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00305bb59f84487b2afcdc30910a663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.683878 2052 tablet_service.cc:1467] Processing CreateTablet for tablet c00305bb59f84487b2afcdc30910a663 (DEFAULT_TABLE table=rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.684303 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00305bb59f84487b2afcdc30910a663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.684552 1792 tablet_service.cc:1467] Processing CreateTablet for tablet c00305bb59f84487b2afcdc30910a663 (DEFAULT_TABLE table=rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:24.684965 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00305bb59f84487b2afcdc30910a663. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:24.699028 4322 tablet_bootstrap.cc:492] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:24.701571 4322 tablet_bootstrap.cc:654] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.707417 4323 tablet_bootstrap.cc:492] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:24.710906 4323 tablet_bootstrap.cc:654] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.711174 4324 tablet_bootstrap.cc:492] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:24.711896 4322 tablet_bootstrap.cc:492] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:24.712059 4322 ts_tablet_manager.cc:1367] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 08:12:24.712883 4322 raft_consensus.cc:348] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.713065 4322 raft_consensus.cc:374] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.713191 4322 raft_consensus.cc:729] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.713418 4322 consensus_queue.cc:260] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.715397 4322 ts_tablet_manager.cc:1398] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:24.715605 4324 tablet_bootstrap.cc:654] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.715981 4322 tablet_bootstrap.cc:492] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:24.717497 4321 tablet_bootstrap.cc:492] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:24.720417 4321 tablet_bootstrap.cc:654] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.724602 4324 tablet_bootstrap.cc:492] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:24.724956 4324 ts_tablet_manager.cc:1367] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20240503 08:12:24.726172 4324 raft_consensus.cc:348] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.726586 4324 raft_consensus.cc:374] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.726753 4324 raft_consensus.cc:729] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.727137 4324 consensus_queue.cc:260] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.728145 4324 ts_tablet_manager.cc:1398] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:24.729806 4322 tablet_bootstrap.cc:654] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.734910 4321 tablet_bootstrap.cc:492] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:24.735261 4321 ts_tablet_manager.cc:1367] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20240503 08:12:24.736944 4321 raft_consensus.cc:348] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.737325 4321 raft_consensus.cc:374] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.737484 4321 raft_consensus.cc:729] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.737887 4321 consensus_queue.cc:260] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.738804 4321 ts_tablet_manager.cc:1398] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:24.739280 4321 tablet_bootstrap.cc:492] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:24.742208 4321 tablet_bootstrap.cc:654] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.743700 4323 tablet_bootstrap.cc:492] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:24.743995 4323 ts_tablet_manager.cc:1367] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.018s
I20240503 08:12:24.745015 4323 raft_consensus.cc:348] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:24.745267 4323 raft_consensus.cc:374] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.745407 4323 raft_consensus.cc:729] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.745710 4323 consensus_queue.cc:260] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:24.746675 4323 ts_tablet_manager.cc:1398] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:24.747181 4323 tablet_bootstrap.cc:492] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:24.749855 4323 tablet_bootstrap.cc:654] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.754148 4322 tablet_bootstrap.cc:492] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:24.754448 4322 ts_tablet_manager.cc:1367] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.004s
I20240503 08:12:24.754984 4321 tablet_bootstrap.cc:492] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:24.755406 4322 raft_consensus.cc:348] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.755676 4322 raft_consensus.cc:374] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.755774 4322 raft_consensus.cc:729] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.756060 4322 consensus_queue.cc:260] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.756635 4323 tablet_bootstrap.cc:492] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:24.756734 4322 ts_tablet_manager.cc:1398] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:24.756886 4323 ts_tablet_manager.cc:1367] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240503 08:12:24.757150 4321 ts_tablet_manager.cc:1367] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20240503 08:12:24.757861 4323 raft_consensus.cc:348] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.758163 4323 raft_consensus.cc:374] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.758306 4323 raft_consensus.cc:729] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.758404 4321 raft_consensus.cc:348] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:24.758644 4323 consensus_queue.cc:260] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.759428 4323 ts_tablet_manager.cc:1398] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:24.759992 4323 tablet_bootstrap.cc:492] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:24.760352 4321 raft_consensus.cc:374] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.760505 4321 raft_consensus.cc:729] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.760617 4322 tablet_bootstrap.cc:492] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:24.760926 4321 consensus_queue.cc:260] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:24.761574 4321 ts_tablet_manager.cc:1398] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:24.762032 4323 tablet_bootstrap.cc:654] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.762264 4322 tablet_bootstrap.cc:654] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:24.765585 4226 raft_consensus.cc:482] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:24.765761 4226 raft_consensus.cc:504] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.766629 4226 leader_election.cc:288] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:24.767175 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668fe552878f42bbb5156c5e162b2630" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:24.767460 2073 raft_consensus.cc:2454] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:24.767616 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668fe552878f42bbb5156c5e162b2630" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:24.767951 1747 leader_election.cc:302] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:24.767964 1944 raft_consensus.cc:2454] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:24.768289 4226 raft_consensus.cc:2781] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:24.768427 4226 raft_consensus.cc:482] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:24.768553 4226 raft_consensus.cc:3037] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.771472 4226 raft_consensus.cc:504] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.772722 4226 leader_election.cc:288] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:24.772861 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668fe552878f42bbb5156c5e162b2630" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:24.773097 2073 raft_consensus.cc:3037] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.775300 2073 raft_consensus.cc:2454] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:24.775839 1747 leader_election.cc:302] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:24.776178 4226 raft_consensus.cc:2781] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:24.776575 4226 raft_consensus.cc:686] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:24.776949 4226 consensus_queue.cc:237] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.777503 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668fe552878f42bbb5156c5e162b2630" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:24.777808 1944 raft_consensus.cc:3037] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.785607 4323 tablet_bootstrap.cc:492] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:24.785791 4323 ts_tablet_manager.cc:1367] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.009s
I20240503 08:12:24.786605 4323 raft_consensus.cc:348] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.786834 4323 raft_consensus.cc:374] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.786934 4323 raft_consensus.cc:729] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.787209 4323 consensus_queue.cc:260] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.788242 4323 ts_tablet_manager.cc:1398] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:24.792569 4266 raft_consensus.cc:482] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:24.792765 4266 raft_consensus.cc:504] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.793778 4266 leader_election.cc:288] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:24.795393 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbf914fa37043e8874583d1fdc914b6" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:24.795389 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbf914fa37043e8874583d1fdc914b6" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:24.795644 2072 raft_consensus.cc:2454] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:24.795750 2203 raft_consensus.cc:2454] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:24.797593 1747 leader_election.cc:302] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:24.797897 4266 raft_consensus.cc:2781] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:24.798023 4266 raft_consensus.cc:482] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:24.798123 4266 raft_consensus.cc:3037] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.798914 1944 raft_consensus.cc:2454] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:24.800565 4322 tablet_bootstrap.cc:492] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:24.800714 4322 ts_tablet_manager.cc:1367] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.013s
I20240503 08:12:24.801211 4266 raft_consensus.cc:504] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.801312 1669 catalog_manager.cc:5484] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:24.801605 4322 raft_consensus.cc:348] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:24.801800 4322 raft_consensus.cc:374] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:24.801901 4322 raft_consensus.cc:729] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:24.801965 4266 leader_election.cc:288] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:24.802132 4322 consensus_queue.cc:260] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:24.802670 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbf914fa37043e8874583d1fdc914b6" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:24.802874 2072 raft_consensus.cc:3037] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.805806 2072 raft_consensus.cc:2454] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:24.806541 4322 ts_tablet_manager.cc:1398] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:24.808009 1747 leader_election.cc:302] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:24.808480 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbf914fa37043e8874583d1fdc914b6" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:24.808714 2202 raft_consensus.cc:3037] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:24.808841 4335 raft_consensus.cc:2781] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:24.810643 4335 raft_consensus.cc:686] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:24.810966 4335 consensus_queue.cc:237] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:24.814205 2202 raft_consensus.cc:2454] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:24.817317 1669 catalog_manager.cc:5484] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:25.103225 4138 raft_consensus.cc:482] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:25.103497 4138 raft_consensus.cc:504] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:25.104153 4138 leader_election.cc:288] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:25.104672 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00305bb59f84487b2afcdc30910a663" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:25.104920 1813 raft_consensus.cc:2454] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:25.104962 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00305bb59f84487b2afcdc30910a663" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:25.105221 1942 raft_consensus.cc:2454] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:25.105417 2009 leader_election.cc:302] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:25.105778 4138 raft_consensus.cc:2781] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:25.105886 4138 raft_consensus.cc:482] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:25.106007 4138 raft_consensus.cc:3037] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:25.108044 4138 raft_consensus.cc:504] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:25.108724 4138 leader_election.cc:288] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:25.109319 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00305bb59f84487b2afcdc30910a663" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:25.109391 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00305bb59f84487b2afcdc30910a663" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:25.109575 1813 raft_consensus.cc:3037] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:25.109617 1942 raft_consensus.cc:3037] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:25.112481 1813 raft_consensus.cc:2454] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:25.112478 1942 raft_consensus.cc:2454] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:25.112993 2009 leader_election.cc:302] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:25.113344 4138 raft_consensus.cc:2781] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:25.113591 4138 raft_consensus.cc:686] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:25.113883 4138 consensus_queue.cc:237] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:25.117506 1669 catalog_manager.cc:5484] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:25.181185 4335 consensus_queue.cc:1035] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:25.211782 4266 consensus_queue.cc:1035] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:25.220093 4226 consensus_queue.cc:1035] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:25.243356 4266 consensus_queue.cc:1035] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:25.498705 4134 consensus_queue.cc:1035] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:25.506435 4134 consensus_queue.cc:1035] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:25.587760 4338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:25.588201 4338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:25.597851 4338 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
INSERT report
rows total: 2000
time total: 321.573 ms
time per row: 0.160787 ms
I20240503 08:12:26.100626 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:26.114074 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 913a3aacff964ac9a321fac187df33ee (DEFAULT_TABLE table=rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.114599 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913a3aacff964ac9a321fac187df33ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.114943 2182 tablet_service.cc:1467] Processing CreateTablet for tablet 9e371231b34f45dfac455bbbbe156263 (DEFAULT_TABLE table=rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.115417 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e371231b34f45dfac455bbbbe156263. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.115356 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 913a3aacff964ac9a321fac187df33ee (DEFAULT_TABLE table=rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.115796 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913a3aacff964ac9a321fac187df33ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.115783 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 9e371231b34f45dfac455bbbbe156263 (DEFAULT_TABLE table=rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.116214 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e371231b34f45dfac455bbbbe156263. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.118204 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 9d11cd50b562408f9d3bc24a974bdba8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.118598 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d11cd50b562408f9d3bc24a974bdba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.119864 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 913a3aacff964ac9a321fac187df33ee (DEFAULT_TABLE table=rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.120251 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913a3aacff964ac9a321fac187df33ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.120370 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 9e371231b34f45dfac455bbbbe156263 (DEFAULT_TABLE table=rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.120755 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e371231b34f45dfac455bbbbe156263. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.124222 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 9d11cd50b562408f9d3bc24a974bdba8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.124652 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d11cd50b562408f9d3bc24a974bdba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.133021 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 9d11cd50b562408f9d3bc24a974bdba8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:26.133694 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d11cd50b562408f9d3bc24a974bdba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:26.156858 4406 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:26.157164 4405 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:26.157730 4403 tablet_bootstrap.cc:492] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:26.159777 4403 tablet_bootstrap.cc:654] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.160790 4406 tablet_bootstrap.cc:654] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.164533 4404 tablet_bootstrap.cc:492] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:26.168126 4404 tablet_bootstrap.cc:654] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.170305 4405 tablet_bootstrap.cc:654] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.175503 4404 tablet_bootstrap.cc:492] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:26.175735 4404 ts_tablet_manager.cc:1367] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20240503 08:12:26.178071 4403 tablet_bootstrap.cc:492] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:26.178112 4404 raft_consensus.cc:348] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.178342 4403 ts_tablet_manager.cc:1367] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.009s
I20240503 08:12:26.179225 4406 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:26.179186 4403 raft_consensus.cc:348] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.179456 4406 ts_tablet_manager.cc:1367] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240503 08:12:26.179510 4403 raft_consensus.cc:374] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.179656 4403 raft_consensus.cc:729] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.179998 4403 consensus_queue.cc:260] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.180732 4403 ts_tablet_manager.cc:1398] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:26.181362 4403 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:26.182999 4403 tablet_bootstrap.cc:654] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.180313 4406 raft_consensus.cc:348] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.183929 4406 raft_consensus.cc:374] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.184056 4406 raft_consensus.cc:729] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.184317 4406 consensus_queue.cc:260] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.185026 4406 ts_tablet_manager.cc:1398] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:26.185222 4404 raft_consensus.cc:374] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.185353 4404 raft_consensus.cc:729] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.185623 4404 consensus_queue.cc:260] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.187372 4406 tablet_bootstrap.cc:492] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:26.187669 4404 ts_tablet_manager.cc:1398] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 08:12:26.188011 4404 tablet_bootstrap.cc:492] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:26.189743 4404 tablet_bootstrap.cc:654] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.191908 4406 tablet_bootstrap.cc:654] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.196965 4405 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:26.197302 4405 ts_tablet_manager.cc:1367] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.000s
I20240503 08:12:26.198544 4405 raft_consensus.cc:348] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.199003 4405 raft_consensus.cc:374] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.199177 4405 raft_consensus.cc:729] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.199584 4405 consensus_queue.cc:260] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.200403 4405 ts_tablet_manager.cc:1398] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:26.202273 4405 tablet_bootstrap.cc:492] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:26.207172 4405 tablet_bootstrap.cc:654] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.216878 4403 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:26.217233 4403 ts_tablet_manager.cc:1367] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.021s
I20240503 08:12:26.218420 4403 raft_consensus.cc:348] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.218734 4403 raft_consensus.cc:374] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.218844 4403 raft_consensus.cc:729] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.219208 4403 consensus_queue.cc:260] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.221773 4403 ts_tablet_manager.cc:1398] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:26.235774 4406 tablet_bootstrap.cc:492] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:26.236081 4406 ts_tablet_manager.cc:1367] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.009s
I20240503 08:12:26.236981 4404 tablet_bootstrap.cc:492] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:26.237216 4404 ts_tablet_manager.cc:1367] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.049s user 0.029s sys 0.015s
I20240503 08:12:26.237239 4406 raft_consensus.cc:348] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.237548 4406 raft_consensus.cc:374] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.237677 4406 raft_consensus.cc:729] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.237840 4405 tablet_bootstrap.cc:492] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:26.238096 4405 ts_tablet_manager.cc:1367] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.000s
I20240503 08:12:26.238039 4406 consensus_queue.cc:260] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.238121 4404 raft_consensus.cc:348] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.238422 4404 raft_consensus.cc:374] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.238556 4404 raft_consensus.cc:729] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.238864 4404 consensus_queue.cc:260] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.239152 4405 raft_consensus.cc:348] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.239405 4405 raft_consensus.cc:374] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.239480 4405 raft_consensus.cc:729] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.241786 4406 ts_tablet_manager.cc:1398] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:26.242133 4404 ts_tablet_manager.cc:1398] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:26.243891 4405 consensus_queue.cc:260] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.247994 4405 ts_tablet_manager.cc:1398] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:12:26.248564 4405 tablet_bootstrap.cc:492] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:26.250176 4405 tablet_bootstrap.cc:654] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:26.250696 4261 raft_consensus.cc:482] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:26.250882 4261 raft_consensus.cc:504] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.251637 4261 leader_election.cc:288] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:26.252343 4405 tablet_bootstrap.cc:492] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:26.252377 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d11cd50b562408f9d3bc24a974bdba8" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:26.252506 4405 ts_tablet_manager.cc:1367] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:26.252418 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d11cd50b562408f9d3bc24a974bdba8" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:26.252640 2073 raft_consensus.cc:2454] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
W20240503 08:12:26.253243 1878 leader_election.cc:341] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:26.253319 4405 raft_consensus.cc:348] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.253545 4405 raft_consensus.cc:374] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:26.253687 4405 raft_consensus.cc:729] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:26.253785 1878 leader_election.cc:302] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters: 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:26.253968 4405 consensus_queue.cc:260] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.254441 4142 raft_consensus.cc:2781] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:26.254555 4142 raft_consensus.cc:482] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:26.254678 4142 raft_consensus.cc:3037] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.254778 4405 ts_tablet_manager.cc:1398] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:26.257135 4142 raft_consensus.cc:504] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.257702 4142 leader_election.cc:288] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:26.258299 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d11cd50b562408f9d3bc24a974bdba8" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:26.258502 2073 raft_consensus.cc:3037] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.261518 2073 raft_consensus.cc:2454] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:26.262123 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d11cd50b562408f9d3bc24a974bdba8" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:26.262367 2203 raft_consensus.cc:3037] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.265429 2203 raft_consensus.cc:2454] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:26.265930 1878 leader_election.cc:302] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:26.267495 4142 raft_consensus.cc:2781] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:26.267709 4142 raft_consensus.cc:686] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:26.268025 4142 consensus_queue.cc:237] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.273464 1665 catalog_manager.cc:5484] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:26.412648 4294 raft_consensus.cc:482] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:26.412876 4294 raft_consensus.cc:504] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.413642 4294 leader_election.cc:288] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:26.414122 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913a3aacff964ac9a321fac187df33ee" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:26.414148 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913a3aacff964ac9a321fac187df33ee" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:26.414335 1942 raft_consensus.cc:2454] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:26.414376 1811 raft_consensus.cc:2454] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:26.414986 2139 leader_election.cc:302] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:26.415299 4294 raft_consensus.cc:2781] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:26.415464 4294 raft_consensus.cc:482] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:26.415563 4294 raft_consensus.cc:3037] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.417963 4294 raft_consensus.cc:504] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.418545 4294 leader_election.cc:288] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:26.419206 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913a3aacff964ac9a321fac187df33ee" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:26.419464 1811 raft_consensus.cc:3037] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.419638 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913a3aacff964ac9a321fac187df33ee" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:26.419863 1944 raft_consensus.cc:3037] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.422461 1811 raft_consensus.cc:2454] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:26.422659 1944 raft_consensus.cc:2454] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:26.423007 2139 leader_election.cc:302] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:26.423380 4294 raft_consensus.cc:2781] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:26.423771 4294 raft_consensus.cc:686] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:26.424095 4294 consensus_queue.cc:237] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.424157 4393 raft_consensus.cc:482] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:26.424314 4393 raft_consensus.cc:504] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.425066 4393 leader_election.cc:288] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:26.425913 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e371231b34f45dfac455bbbbe156263" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:26.426082 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e371231b34f45dfac455bbbbe156263" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:26.426246 2073 raft_consensus.cc:2454] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:26.426395 2202 raft_consensus.cc:2454] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:26.427608 1747 leader_election.cc:302] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:26.428122 4393 raft_consensus.cc:2781] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:26.428267 4393 raft_consensus.cc:482] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:26.428391 4393 raft_consensus.cc:3037] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.431161 4393 raft_consensus.cc:504] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.431937 4393 leader_election.cc:288] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:26.431967 1665 catalog_manager.cc:5484] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:26.432564 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e371231b34f45dfac455bbbbe156263" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:26.432562 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e371231b34f45dfac455bbbbe156263" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:26.432812 2073 raft_consensus.cc:3037] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.432857 2203 raft_consensus.cc:3037] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:26.436023 2073 raft_consensus.cc:2454] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:26.436517 1747 leader_election.cc:302] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:26.436591 2203 raft_consensus.cc:2454] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:26.436863 4393 raft_consensus.cc:2781] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:26.437019 4393 raft_consensus.cc:686] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:26.437319 4393 consensus_queue.cc:237] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:26.440486 1665 catalog_manager.cc:5484] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:26.763670 4142 consensus_queue.cc:1035] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:26.780608 4261 consensus_queue.cc:1035] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:26.843880 4420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:26.844233 4420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:26.853583 4420 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:12:26.869174 4393 consensus_queue.cc:1035] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:26.882001 4393 consensus_queue.cc:1035] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_19'
I20240503 08:12:26.963675 4417 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:26.980202 4294 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:27.047251 2139 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.194:41975, user_credentials={real_user=slave}} blocked reactor thread for 54470.9us
INSERT report
rows total: 2000
time total: 344.932 ms
time per row: 0.172466 ms
I20240503 08:12:27.367686 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.197:42465
--local_ip_for_outbound_sockets=127.27.157.197
--tserver_master_addrs=127.27.157.254:37795
--webserver_port=37373
--webserver_interface=127.27.157.197
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:12:27.847900 4485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:27.848222 4485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:27.848331 4485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:27.856403 4485 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:27.856545 4485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:27.856667 4485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.197
I20240503 08:12:27.866570 4485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.197:42465
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.157.197
--webserver_port=37373
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.197
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:27.867960 4485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:27.869907 4485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:27.886065 4491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:27.886571 4492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:27.889750 4494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:27.890619 4485 server_base.cc:1034] running on GCE node
I20240503 08:12:27.891376 4485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:27.892648 4485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:27.893909 4485 hybrid_clock.cc:648] HybridClock initialized: now 1714723947893810 us; error 87 us; skew 500 ppm
I20240503 08:12:27.894354 4485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:27.897337 4485 webserver.cc:460] Webserver started at http://127.27.157.197:37373/ using document root <none> and password file <none>
I20240503 08:12:27.898298 4485 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:27.898463 4485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:27.904688 4485 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:12:27.907960 4500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:27.909871 4485 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:12:27.910051 4485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5f95b05d111e4090bbc59dec394ed0ab"
format_stamp: "Formatted at 2024-05-03 08:11:55 on dist-test-slave-dj8w"
I20240503 08:12:27.910914 4485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:27.928565 4485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:27.929420 4485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:27.929737 4485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:27.930811 4485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:27.932510 4485 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:27.932605 4485 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:27.932736 4485 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:27.932793 4485 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:27.968554 4485 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.197:42465
I20240503 08:12:27.968631 4613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.197:42465 every 8 connection(s)
I20240503 08:12:27.970124 4485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:12:27.970985 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 4485
I20240503 08:12:27.971866 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.198:46559
--local_ip_for_outbound_sockets=127.27.157.198
--tserver_master_addrs=127.27.157.254:37795
--webserver_port=42091
--webserver_interface=127.27.157.198
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:27.986042 4614 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:12:27.986424 4614 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:27.987752 4614 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:12:27.989711 1665 ts_manager.cc:194] Re-registered known tserver with Master: 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:27.990653 1665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.197:53175
W20240503 08:12:28.370064 4618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:28.370481 4618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:28.370666 4618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:28.379190 4618 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:28.379403 4618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:28.379608 4618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.198
I20240503 08:12:28.391479 4618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.198:46559
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.157.198
--webserver_port=42091
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.198
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:28.392874 4618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:28.394804 4618 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:28.408763 4625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:28.409242 4626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:28.410212 4628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:28.411278 4618 server_base.cc:1034] running on GCE node
I20240503 08:12:28.411962 4618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:28.412977 4618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:28.414186 4618 hybrid_clock.cc:648] HybridClock initialized: now 1714723948414121 us; error 57 us; skew 500 ppm
I20240503 08:12:28.414501 4618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:28.423401 4618 webserver.cc:460] Webserver started at http://127.27.157.198:42091/ using document root <none> and password file <none>
I20240503 08:12:28.424310 4618 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:28.424460 4618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:28.431396 4618 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 08:12:28.434779 4634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:28.436444 4618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 08:12:28.436620 4618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "31f41542452b4b1eb994ff4223d039fc"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:12:28.437552 4618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:28.458896 4618 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:28.459862 4618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:28.460155 4618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:28.461287 4618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:28.463063 4618 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:28.463162 4618 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:28.463294 4618 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:28.463372 4618 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240503 08:12:28.501475 4618 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.198:46559
I20240503 08:12:28.501550 4747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.198:46559 every 8 connection(s)
I20240503 08:12:28.503224 4618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:12:28.512684 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 4618
I20240503 08:12:28.513427 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.199:39455
--local_ip_for_outbound_sockets=127.27.157.199
--tserver_master_addrs=127.27.157.254:37795
--webserver_port=41391
--webserver_interface=127.27.157.199
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:28.518934 4748 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:12:28.519289 4748 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:28.520314 4748 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:12:28.521858 1665 ts_manager.cc:194] Re-registered known tserver with Master: 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:28.522692 1665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.198:47081
W20240503 08:12:28.905910 4752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:28.906324 4752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:28.906472 4752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:28.914836 4752 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:28.914984 4752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:28.915140 4752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.199
I20240503 08:12:28.925808 4752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.199:39455
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.157.199
--webserver_port=41391
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.199
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:28.927266 4752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:28.929464 4752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:28.942909 4758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:28.943619 4757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:28.944444 4760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:28.945518 4752 server_base.cc:1034] running on GCE node
I20240503 08:12:28.946295 4752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:28.947595 4752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:28.948834 4752 hybrid_clock.cc:648] HybridClock initialized: now 1714723948948772 us; error 59 us; skew 500 ppm
I20240503 08:12:28.949270 4752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:28.952260 4752 webserver.cc:460] Webserver started at http://127.27.157.199:41391/ using document root <none> and password file <none>
I20240503 08:12:28.953058 4752 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:28.953193 4752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:28.959779 4752 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 08:12:28.962958 4766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:28.964625 4752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:28.964821 4752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f24648c92c384f1ba349eee801f985e2"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:12:28.965760 4752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:28.987931 4752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:28.988826 4752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:28.989122 4752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:28.990394 4752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:28.992259 4752 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:28.992344 4752 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:28.992487 4752 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:28.992550 4752 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:28.995425 4614 heartbeater.cc:503] Master 127.27.157.254:37795 was elected leader, sending a full tablet report...
I20240503 08:12:29.038355 4752 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.199:39455
I20240503 08:12:29.038403 4879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.199:39455 every 8 connection(s)
I20240503 08:12:29.040297 4752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:12:29.044287 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 4752
I20240503 08:12:29.045104 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.200:45283
--local_ip_for_outbound_sockets=127.27.157.200
--tserver_master_addrs=127.27.157.254:37795
--webserver_port=34781
--webserver_interface=127.27.157.200
--builtin_ntp_servers=127.27.157.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:12:29.055923 4880 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:12:29.056342 4880 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:29.057500 4880 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:12:29.059399 1665 ts_manager.cc:194] Re-registered known tserver with Master: f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:29.060335 1665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.199:45959
W20240503 08:12:29.440356 4884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:29.440721 4884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:12:29.440886 4884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:29.450349 4884 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:12:29.450491 4884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:12:29.450680 4884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.200
I20240503 08:12:29.461040 4884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.200:45283
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.157.200
--webserver_port=34781
--tserver_master_addrs=127.27.157.254:37795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.200
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:12:29.462533 4884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:12:29.464471 4884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:29.478924 4891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:29.479672 4893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:29.480252 4890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:29.481830 4884 server_base.cc:1034] running on GCE node
I20240503 08:12:29.482533 4884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:12:29.483666 4884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:12:29.484886 4884 hybrid_clock.cc:648] HybridClock initialized: now 1714723949484820 us; error 65 us; skew 500 ppm
I20240503 08:12:29.485211 4884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:12:29.487685 4884 webserver.cc:460] Webserver started at http://127.27.157.200:34781/ using document root <none> and password file <none>
I20240503 08:12:29.488461 4884 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:29.488586 4884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:29.495491 4884 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20240503 08:12:29.499275 4899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:29.501201 4884 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:29.501397 4884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "00b61625d8a7460892c118a2570afda5"
format_stamp: "Formatted at 2024-05-03 08:11:56 on dist-test-slave-dj8w"
I20240503 08:12:29.502457 4884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:29.523887 4884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:29.524794 4884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:12:29.525075 4884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:29.526010 4748 heartbeater.cc:503] Master 127.27.157.254:37795 was elected leader, sending a full tablet report...
I20240503 08:12:29.526288 4884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:29.529162 4884 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:29.529260 4884 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:29.529368 4884 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:29.529412 4884 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:29.569402 4884 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.200:45283
I20240503 08:12:29.569444 5012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.200:45283 every 8 connection(s)
I20240503 08:12:29.571370 4884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:12:29.575642 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 4884
I20240503 08:12:29.582576 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:29.592630 5013 heartbeater.cc:347] Connected to a master server at 127.27.157.254:37795
I20240503 08:12:29.593019 5013 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:29.594455 5013 heartbeater.cc:511] Master 127.27.157.254:37795 requested a full tablet report, sending...
I20240503 08:12:29.596629 1670 ts_manager.cc:194] Re-registered known tserver with Master: 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:29.598325 1670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.200:43103
I20240503 08:12:29.605140 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff08610d0b9400d83ab22a36bc43d6e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c4d5c8d327c4e179bb37f70a69a6696]), partition=
I20240503 08:12:29.605592 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff08610d0b9400d83ab22a36bc43d6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:29.623900 5030 tablet_bootstrap.cc:492] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:29.626157 5030 tablet_bootstrap.cc:654] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:29.628934 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff08610d0b9400d83ab22a36bc43d6e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c4d5c8d327c4e179bb37f70a69a6696]), partition=
I20240503 08:12:29.628934 4549 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff08610d0b9400d83ab22a36bc43d6e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c4d5c8d327c4e179bb37f70a69a6696]), partition=
I20240503 08:12:29.630139 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff08610d0b9400d83ab22a36bc43d6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:29.630299 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff08610d0b9400d83ab22a36bc43d6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:29.631162 5030 tablet_bootstrap.cc:492] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:29.631377 5030 ts_tablet_manager.cc:1367] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 08:12:29.632216 5030 raft_consensus.cc:348] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:29.632436 5030 raft_consensus.cc:374] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:29.632513 5030 raft_consensus.cc:729] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:29.632721 5030 consensus_queue.cc:260] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:29.634828 5030 ts_tablet_manager.cc:1398] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:29.641690 4619 raft_consensus.cc:482] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:29.641873 4619 raft_consensus.cc:504] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:29.642889 4619 leader_election.cc:288] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
W20240503 08:12:29.664530 1749 leader_election.cc:341] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Not found: Tablet not found: 4ff08610d0b9400d83ab22a36bc43d6e
I20240503 08:12:29.665896 5034 tablet_bootstrap.cc:492] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:29.673207 5034 tablet_bootstrap.cc:654] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:29.674649 5034 log.cc:824] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:29.679594 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff08610d0b9400d83ab22a36bc43d6e" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
W20240503 08:12:29.684029 1746 leader_election.cc:341] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:29.684177 1746 leader_election.cc:302] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f581390da449ccb9e299011296405e; no voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2
I20240503 08:12:29.684571 4436 raft_consensus.cc:2726] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:29.689139 5034 tablet_bootstrap.cc:492] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:29.689527 5034 ts_tablet_manager.cc:1367] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.006s
I20240503 08:12:29.695791 5036 tablet_bootstrap.cc:492] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:29.697482 5034 raft_consensus.cc:348] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:29.698015 5034 raft_consensus.cc:374] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:29.698144 5034 raft_consensus.cc:729] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:29.699008 5034 consensus_queue.cc:260] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:29.701536 5036 tablet_bootstrap.cc:654] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:29.702113 5034 ts_tablet_manager.cc:1398] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20240503 08:12:29.702941 5036 log.cc:824] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:29.706221 5036 tablet_bootstrap.cc:492] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:29.706535 5036 ts_tablet_manager.cc:1367] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 08:12:29.715234 5036 raft_consensus.cc:348] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:29.715765 5036 raft_consensus.cc:374] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:29.715880 5036 raft_consensus.cc:729] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:29.716519 5036 consensus_queue.cc:260] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:29.717722 4880 heartbeater.cc:503] Master 127.27.157.254:37795 was elected leader, sending a full tablet report...
I20240503 08:12:29.718199 5036 ts_tablet_manager.cc:1398] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20240503 08:12:29.723305 4615 tablet.cc:2369] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:29.737493 5037 raft_consensus.cc:482] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:29.737779 5037 raft_consensus.cc:504] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:29.739806 5037 leader_election.cc:288] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:29.773327 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff08610d0b9400d83ab22a36bc43d6e" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:29.773972 4835 raft_consensus.cc:2454] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:29.774716 4504 leader_election.cc:302] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:29.775346 5037 raft_consensus.cc:2781] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:29.775499 5037 raft_consensus.cc:482] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:29.775600 5037 raft_consensus.cc:3037] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:29.779175 5037 raft_consensus.cc:504] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:29.780138 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff08610d0b9400d83ab22a36bc43d6e" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:29.780400 1813 raft_consensus.cc:2454] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:29.780438 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff08610d0b9400d83ab22a36bc43d6e" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:29.780408 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff08610d0b9400d83ab22a36bc43d6e" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:29.780676 4835 raft_consensus.cc:3037] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:29.780853 1812 raft_consensus.cc:3037] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:29.780865 5037 leader_election.cc:288] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:29.783361 4835 raft_consensus.cc:2454] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:29.783823 1812 raft_consensus.cc:2454] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:29.783854 4504 leader_election.cc:302] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:29.784363 5037 raft_consensus.cc:2781] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:29.784924 5037 raft_consensus.cc:686] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:29.785522 5037 consensus_queue.cc:237] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:29.791723 1665 catalog_manager.cc:5484] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:29.792390 4881 tablet.cc:2369] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:29.910192 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:29.922647 1812 raft_consensus.cc:1261] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:29.923591 4835 raft_consensus.cc:1261] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:29.924077 5037 consensus_queue.cc:1035] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:29.924515 5043 consensus_queue.cc:1035] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:29.939738 1665 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet 4ff08610d0b9400d83ab22a36bc43d6e (table rebalancer_dynamic_table [id=9c4d5c8d327c4e179bb37f70a69a6696]). Expected version 1 got 0
I20240503 08:12:29.942734 1665 catalog_manager.cc:5576] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181) does not have the latest schema for tablet 4ff08610d0b9400d83ab22a36bc43d6e (table rebalancer_dynamic_table [id=9c4d5c8d327c4e179bb37f70a69a6696]). Expected version 1 got 0
I20240503 08:12:29.942855 5046 mvcc.cc:205] Tried to move back new op lower bound from 7023509298874089472 to 7023509298325872640. Current Snapshot: MvccSnapshot[applied={T|T < 7023509298874089472}]
I20240503 08:12:29.946532 5049 mvcc.cc:205] Tried to move back new op lower bound from 7023509298874089472 to 7023509298325872640. Current Snapshot: MvccSnapshot[applied={T|T < 7023509298874089472}]
I20240503 08:12:29.948544 5053 tablet.cc:1722] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:29.950273 1665 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet 4ff08610d0b9400d83ab22a36bc43d6e (table rebalancer_dynamic_table [id=9c4d5c8d327c4e179bb37f70a69a6696]). Expected version 1 got 0
I20240503 08:12:29.951057 5047 mvcc.cc:205] Tried to move back new op lower bound from 7023509298874089472 to 7023509298325872640. Current Snapshot: MvccSnapshot[applied={T|T < 7023509298874089472}]
I20240503 08:12:29.952999 5053 tablet.cc:1620] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:29.955345 5052 tablet.cc:1722] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:29.955657 5054 tablet.cc:1722] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:29.956985 5054 tablet.cc:1620] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:29.957003 1665 catalog_manager.cc:5576] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181) does not have the latest schema for tablet 4ff08610d0b9400d83ab22a36bc43d6e (table rebalancer_dynamic_table [id=9c4d5c8d327c4e179bb37f70a69a6696]). Expected version 1 got 0
I20240503 08:12:29.960232 5052 tablet.cc:1620] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:29.966769 1670 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=9c4d5c8d327c4e179bb37f70a69a6696] alter complete (version 1)
I20240503 08:12:30.069339 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:30.078639 1670 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 08:12:30.078912 1670 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 08:12:30.079622 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:30.080142 5052 tablet.cc:1722] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:30.080675 5052 tablet.cc:1620] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:30.082944 5054 tablet.cc:1722] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:30.083503 5054 tablet.cc:1620] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:30.084239 5053 tablet.cc:1722] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:30.084657 5053 tablet.cc:1620] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:30.097295 1670 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet a336330f849f4aca8d128a2647969f4e
I20240503 08:12:30.098455 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet a336330f849f4aca8d128a2647969f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:30.099239 5060 tablet_replica.cc:330] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:30.099717 5060 raft_consensus.cc:2229] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.100131 5060 raft_consensus.cc:2258] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.101727 5060 ts_tablet_manager.cc:1875] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.109963 1664 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=9c4d5c8d327c4e179bb37f70a69a6696] alter complete (version 2)
I20240503 08:12:30.111515 5060 ts_tablet_manager.cc:1888] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.111747 5060 log.cc:1196] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/a336330f849f4aca8d128a2647969f4e
I20240503 08:12:30.112246 5060 ts_tablet_manager.cc:1909] T a336330f849f4aca8d128a2647969f4e P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:30.114250 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet a336330f849f4aca8d128a2647969f4e (table rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]) successfully deleted
I20240503 08:12:30.119467 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 820ecfef09a34327a08f7c3a893ccb05 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.121209 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820ecfef09a34327a08f7c3a893ccb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.122671 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 820ecfef09a34327a08f7c3a893ccb05 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.123488 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820ecfef09a34327a08f7c3a893ccb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.128167 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.128660 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.130077 1670 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 31e125a3957b4a6298e5b2d7cae30bb9
I20240503 08:12:30.131489 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet a336330f849f4aca8d128a2647969f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:30.131733 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet a336330f849f4aca8d128a2647969f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:30.132191 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e125a3957b4a6298e5b2d7cae30bb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:30.132876 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e125a3957b4a6298e5b2d7cae30bb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:30.133509 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e125a3957b4a6298e5b2d7cae30bb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:30.134747 1670 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 47800c04e5a74c548ce68cb42f74d78f
I20240503 08:12:30.135582 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 47800c04e5a74c548ce68cb42f74d78f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:30.136278 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet 47800c04e5a74c548ce68cb42f74d78f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:30.137594 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 47800c04e5a74c548ce68cb42f74d78f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:30.137866 5060 tablet_replica.cc:330] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:30.138115 5060 raft_consensus.cc:2229] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.138347 5060 raft_consensus.cc:2258] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.139156 5060 ts_tablet_manager.cc:1875] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.145282 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 8737ef0936c641ccaf0b804b1d9c3e07 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.146006 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8737ef0936c641ccaf0b804b1d9c3e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.152209 5036 tablet_bootstrap.cc:492] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:30.154502 5036 tablet_bootstrap.cc:654] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.163486 5060 ts_tablet_manager.cc:1888] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.164242 5036 tablet_bootstrap.cc:492] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:30.164460 5036 ts_tablet_manager.cc:1367] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 08:12:30.164698 5060 log.cc:1196] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/47800c04e5a74c548ce68cb42f74d78f
I20240503 08:12:30.165588 5036 raft_consensus.cc:348] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.165841 5036 raft_consensus.cc:374] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.165953 5036 raft_consensus.cc:729] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.166306 5036 consensus_queue.cc:260] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.166457 5060 ts_tablet_manager.cc:1909] T 47800c04e5a74c548ce68cb42f74d78f P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:30.166879 5036 ts_tablet_manager.cc:1398] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:30.179281 5068 tablet_replica.cc:330] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:30.180109 5069 tablet_bootstrap.cc:492] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:30.181267 4947 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.183555 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.190733 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 8737ef0936c641ccaf0b804b1d9c3e07 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.191022 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 820ecfef09a34327a08f7c3a893ccb05 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.191354 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8737ef0936c641ccaf0b804b1d9c3e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.191737 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820ecfef09a34327a08f7c3a893ccb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.185809 5068 raft_consensus.cc:2229] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.208303 5068 raft_consensus.cc:2258] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.209300 5067 tablet_replica.cc:330] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:30.213145 1671 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:30.215391 5073 tablet_bootstrap.cc:492] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:30.223030 5073 tablet_bootstrap.cc:654] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.227098 5052 tablet.cc:1722] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:30.227810 5066 tablet_replica.cc:330] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:30.227983 5073 log.cc:824] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:30.232029 5053 tablet.cc:1722] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:30.237182 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 47800c04e5a74c548ce68cb42f74d78f (table rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]) successfully deleted
W20240503 08:12:30.236037 2008 consensus_peers.cc:485] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a336330f849f4aca8d128a2647969f4e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:30.238934 5066 raft_consensus.cc:2229] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:30.239934 5066 raft_consensus.cc:2258] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.246096 1670 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=9c4d5c8d327c4e179bb37f70a69a6696] alter complete (version 3)
W20240503 08:12:30.252094 1747 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.195:43117, user_credentials={real_user=slave}} blocked reactor thread for 67490us
I20240503 08:12:30.252457 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.253767 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 8737ef0936c641ccaf0b804b1d9c3e07 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:30.255177 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8737ef0936c641ccaf0b804b1d9c3e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.258596 5054 tablet.cc:1722] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:30.259250 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.261970 5067 raft_consensus.cc:2229] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.262589 5067 raft_consensus.cc:2258] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.264569 5067 ts_tablet_manager.cc:1875] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.268385 5069 tablet_bootstrap.cc:654] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.270859 5068 ts_tablet_manager.cc:1875] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.272773 5066 ts_tablet_manager.cc:1875] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.282709 5074 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:30.284757 5079 tablet_bootstrap.cc:492] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:30.287278 5074 tablet_bootstrap.cc:654] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
W20240503 08:12:30.289074 1879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.193:41181, user_credentials={real_user=slave}} blocked reactor thread for 70971.4us
I20240503 08:12:30.290529 5039 raft_consensus.cc:482] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:30.290789 5039 raft_consensus.cc:504] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.291425 5079 tablet_bootstrap.cc:654] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.292824 5079 log.cc:824] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc: Log is configured to *not* fsync() on all Append() calls
I20240503 08:12:30.293704 5039 leader_election.cc:288] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:30.304667 5079 tablet_bootstrap.cc:492] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:30.305207 5079 ts_tablet_manager.cc:1367] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240503 08:12:30.305595 5069 tablet_bootstrap.cc:492] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:30.305845 5069 ts_tablet_manager.cc:1367] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.126s user 0.002s sys 0.002s
I20240503 08:12:30.306676 5069 raft_consensus.cc:348] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.307080 5069 raft_consensus.cc:374] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.307253 5069 raft_consensus.cc:729] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.307735 5069 consensus_queue.cc:260] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.309096 5069 ts_tablet_manager.cc:1398] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:30.311408 5069 tablet_bootstrap.cc:492] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:30.313844 5069 tablet_bootstrap.cc:654] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.330178 5079 raft_consensus.cc:348] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:30.331755 5079 raft_consensus.cc:374] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.332183 5079 raft_consensus.cc:729] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.333662 5079 consensus_queue.cc:260] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:30.340507 5069 tablet_bootstrap.cc:492] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:30.340818 5069 ts_tablet_manager.cc:1367] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.002s
I20240503 08:12:30.341754 5069 raft_consensus.cc:348] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:30.342062 5069 raft_consensus.cc:374] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.342180 5069 raft_consensus.cc:729] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.335213 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820ecfef09a34327a08f7c3a893ccb05" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:30.342586 5069 consensus_queue.cc:260] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:30.343312 5069 ts_tablet_manager.cc:1398] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240503 08:12:30.345034 4768 leader_election.cc:341] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:30.345937 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820ecfef09a34327a08f7c3a893ccb05" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:30.346161 2203 raft_consensus.cc:2454] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:30.346729 4768 leader_election.cc:302] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, f24648c92c384f1ba349eee801f985e2; no voters: 00b61625d8a7460892c118a2570afda5
I20240503 08:12:30.347373 5039 raft_consensus.cc:2781] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:30.347553 5039 raft_consensus.cc:482] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:30.347687 5039 raft_consensus.cc:3037] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.353469 1670 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:30.353812 1670 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:30.360503 5039 raft_consensus.cc:504] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.364625 5066 ts_tablet_manager.cc:1888] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.364900 5066 log.cc:1196] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/a336330f849f4aca8d128a2647969f4e
I20240503 08:12:30.365694 5066 ts_tablet_manager.cc:1909] T a336330f849f4aca8d128a2647969f4e P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:30.367611 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet a336330f849f4aca8d128a2647969f4e (table rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]) successfully deleted
I20240503 08:12:30.369365 5066 tablet_replica.cc:330] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:30.369921 5066 raft_consensus.cc:2229] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.370299 5066 raft_consensus.cc:2258] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.371110 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820ecfef09a34327a08f7c3a893ccb05" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:30.371441 2203 raft_consensus.cc:3037] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.371742 5066 ts_tablet_manager.cc:1875] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.378357 1670 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 4ff08610d0b9400d83ab22a36bc43d6e
I20240503 08:12:30.380141 4549 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff08610d0b9400d83ab22a36bc43d6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:30.381160 5091 tablet_replica.cc:330] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:30.381590 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff08610d0b9400d83ab22a36bc43d6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:30.382234 5091 raft_consensus.cc:2229] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:30.383172 5091 raft_consensus.cc:2258] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.383363 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff08610d0b9400d83ab22a36bc43d6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:30.375237 2203 raft_consensus.cc:2454] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:30.385556 4768 leader_election.cc:302] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:30.385905 5066 ts_tablet_manager.cc:1888] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.386072 5066 log.cc:1196] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/31e125a3957b4a6298e5b2d7cae30bb9
I20240503 08:12:30.386341 5093 raft_consensus.cc:2781] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:30.386658 5066 ts_tablet_manager.cc:1909] T 31e125a3957b4a6298e5b2d7cae30bb9 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:30.387944 5039 leader_election.cc:288] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:30.387984 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820ecfef09a34327a08f7c3a893ccb05" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:30.388216 5091 ts_tablet_manager.cc:1875] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:12:30.388986 4768 leader_election.cc:341] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:30.391360 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 31e125a3957b4a6298e5b2d7cae30bb9 (table rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]) successfully deleted
I20240503 08:12:30.393163 5093 raft_consensus.cc:686] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:30.394587 5093 consensus_queue.cc:237] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.396003 5092 tablet_replica.cc:330] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:30.397764 5092 raft_consensus.cc:2229] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.398146 5092 raft_consensus.cc:2258] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.400302 5092 ts_tablet_manager.cc:1875] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.403623 5073 tablet_bootstrap.cc:492] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:30.404034 5073 ts_tablet_manager.cc:1367] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.189s user 0.042s sys 0.066s
I20240503 08:12:30.404086 5068 ts_tablet_manager.cc:1888] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.404268 5068 log.cc:1196] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/31e125a3957b4a6298e5b2d7cae30bb9
I20240503 08:12:30.404844 5068 ts_tablet_manager.cc:1909] T 31e125a3957b4a6298e5b2d7cae30bb9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:30.406582 5068 tablet_replica.cc:330] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:30.406683 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 31e125a3957b4a6298e5b2d7cae30bb9 (table rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]) successfully deleted
I20240503 08:12:30.406912 5074 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:30.407063 5074 ts_tablet_manager.cc:1367] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.000s
I20240503 08:12:30.408066 5074 raft_consensus.cc:348] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.408727 5074 raft_consensus.cc:374] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.408908 5074 raft_consensus.cc:729] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.409336 5074 consensus_queue.cc:260] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.409204 1671 catalog_manager.cc:5484] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:30.410212 5074 ts_tablet_manager.cc:1398] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:30.411412 5067 ts_tablet_manager.cc:1888] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.411592 5067 log.cc:1196] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/a336330f849f4aca8d128a2647969f4e
I20240503 08:12:30.412379 5067 ts_tablet_manager.cc:1909] T a336330f849f4aca8d128a2647969f4e P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:30.414135 5067 tablet_replica.cc:330] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:30.414136 5091 ts_tablet_manager.cc:1888] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:30.414325 5091 log.cc:1196] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/4ff08610d0b9400d83ab22a36bc43d6e
I20240503 08:12:30.414564 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet a336330f849f4aca8d128a2647969f4e (table rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]) successfully deleted
I20240503 08:12:30.418494 5079 ts_tablet_manager.cc:1398] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.113s user 0.021s sys 0.018s
I20240503 08:12:30.419014 5079 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:30.419564 5067 raft_consensus.cc:2229] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:30.420117 5067 raft_consensus.cc:2258] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.422009 5079 tablet_bootstrap.cc:654] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.422937 5068 raft_consensus.cc:2229] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.423216 5068 raft_consensus.cc:2258] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.424357 5067 ts_tablet_manager.cc:1875] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.426335 5073 raft_consensus.cc:348] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.427523 5091 ts_tablet_manager.cc:1909] T 4ff08610d0b9400d83ab22a36bc43d6e P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:30.429425 5073 raft_consensus.cc:374] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.429502 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 4ff08610d0b9400d83ab22a36bc43d6e (table rebalancer_dynamic_table_new_name [id=9c4d5c8d327c4e179bb37f70a69a6696]) successfully deleted
I20240503 08:12:30.429874 5073 raft_consensus.cc:729] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.430771 5073 consensus_queue.cc:260] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.433075 5068 ts_tablet_manager.cc:1875] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.434067 5079 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:30.434300 5079 ts_tablet_manager.cc:1367] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 08:12:30.434998 5092 ts_tablet_manager.cc:1888] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:30.435201 5092 log.cc:1196] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/4ff08610d0b9400d83ab22a36bc43d6e
I20240503 08:12:30.435863 5092 ts_tablet_manager.cc:1909] T 4ff08610d0b9400d83ab22a36bc43d6e P f24648c92c384f1ba349eee801f985e2: Deleting consensus metadata
I20240503 08:12:30.435751 5079 raft_consensus.cc:348] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.437124 5079 raft_consensus.cc:374] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.437283 5079 raft_consensus.cc:729] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.437616 5079 consensus_queue.cc:260] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.439302 5079 ts_tablet_manager.cc:1398] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:30.439363 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet 4ff08610d0b9400d83ab22a36bc43d6e (table rebalancer_dynamic_table_new_name [id=9c4d5c8d327c4e179bb37f70a69a6696]) successfully deleted
I20240503 08:12:30.439723 5068 ts_tablet_manager.cc:1888] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.439890 5068 log.cc:1196] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/47800c04e5a74c548ce68cb42f74d78f
I20240503 08:12:30.440335 5068 ts_tablet_manager.cc:1909] T 47800c04e5a74c548ce68cb42f74d78f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:30.441038 5013 heartbeater.cc:503] Master 127.27.157.254:37795 was elected leader, sending a full tablet report...
W20240503 08:12:30.442390 1747 consensus_peers.cc:485] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47800c04e5a74c548ce68cb42f74d78f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:30.442832 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 47800c04e5a74c548ce68cb42f74d78f (table rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]) successfully deleted
I20240503 08:12:30.445665 5073 ts_tablet_manager.cc:1398] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.041s user 0.003s sys 0.007s
I20240503 08:12:30.446216 5073 tablet_bootstrap.cc:492] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:30.446976 5067 ts_tablet_manager.cc:1888] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.447142 5067 log.cc:1196] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/31e125a3957b4a6298e5b2d7cae30bb9
I20240503 08:12:30.447571 5067 ts_tablet_manager.cc:1909] T 31e125a3957b4a6298e5b2d7cae30bb9 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:30.448361 5073 tablet_bootstrap.cc:654] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.448585 5067 tablet_replica.cc:330] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:30.448655 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 31e125a3957b4a6298e5b2d7cae30bb9 (table rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]) successfully deleted
I20240503 08:12:30.448813 5067 raft_consensus.cc:2229] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:30.449157 5067 raft_consensus.cc:2258] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.449911 5067 ts_tablet_manager.cc:1875] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.456586 5067 ts_tablet_manager.cc:1888] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.456769 5067 log.cc:1196] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/47800c04e5a74c548ce68cb42f74d78f
I20240503 08:12:30.457280 5067 ts_tablet_manager.cc:1909] T 47800c04e5a74c548ce68cb42f74d78f P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:30.457581 5073 tablet_bootstrap.cc:492] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:30.457746 5073 ts_tablet_manager.cc:1367] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240503 08:12:30.458302 5067 tablet_replica.cc:330] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:30.458534 5067 raft_consensus.cc:2229] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.458668 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 47800c04e5a74c548ce68cb42f74d78f (table rebalance_test_table_0 [id=20dfb0814fde4c90b5e77612f7f51f3c]) successfully deleted
I20240503 08:12:30.458786 5067 raft_consensus.cc:2258] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.458837 5073 raft_consensus.cc:348] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:30.459066 5073 raft_consensus.cc:374] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.459167 5073 raft_consensus.cc:729] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.459476 5073 consensus_queue.cc:260] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:30.460459 5073 ts_tablet_manager.cc:1398] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 08:12:30.460750 5073 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:30.460978 5067 ts_tablet_manager.cc:1875] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.463472 5073 tablet_bootstrap.cc:654] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.469005 5067 ts_tablet_manager.cc:1888] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:30.469247 5067 log.cc:1196] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/4ff08610d0b9400d83ab22a36bc43d6e
I20240503 08:12:30.469645 5073 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:30.469758 5067 ts_tablet_manager.cc:1909] T 4ff08610d0b9400d83ab22a36bc43d6e P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:30.469879 5073 ts_tablet_manager.cc:1367] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240503 08:12:30.470536 5073 raft_consensus.cc:348] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.470757 5073 raft_consensus.cc:374] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.470839 5073 raft_consensus.cc:729] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.471030 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 4ff08610d0b9400d83ab22a36bc43d6e (table rebalancer_dynamic_table_new_name [id=9c4d5c8d327c4e179bb37f70a69a6696]) successfully deleted
I20240503 08:12:30.471181 5073 consensus_queue.cc:260] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.472401 5073 ts_tablet_manager.cc:1398] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:30.483745 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:30.492799 4549 tablet_service.cc:1467] Processing CreateTablet for tablet d70c0406fe734040858907f3b68fdd3c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=65d93adde78b4eb098ee656448a6ab57]), partition=
I20240503 08:12:30.493239 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d70c0406fe734040858907f3b68fdd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.493485 4683 tablet_service.cc:1467] Processing CreateTablet for tablet d70c0406fe734040858907f3b68fdd3c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=65d93adde78b4eb098ee656448a6ab57]), partition=
I20240503 08:12:30.493868 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d70c0406fe734040858907f3b68fdd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.499781 1923 tablet_service.cc:1467] Processing CreateTablet for tablet d70c0406fe734040858907f3b68fdd3c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=65d93adde78b4eb098ee656448a6ab57]), partition=
I20240503 08:12:30.499904 4134 raft_consensus.cc:482] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:30.500216 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d70c0406fe734040858907f3b68fdd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:30.500090 4134 raft_consensus.cc:504] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.500926 4134 leader_election.cc:288] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
W20240503 08:12:30.506515 4749 tablet.cc:2369] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:30.509284 5079 tablet_bootstrap.cc:492] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:30.511174 5079 tablet_bootstrap.cc:654] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.527112 5113 tablet_bootstrap.cc:492] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:30.528075 5112 tablet_bootstrap.cc:492] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:30.530055 5112 tablet_bootstrap.cc:654] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.531904 5079 tablet_bootstrap.cc:492] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:30.532166 5079 ts_tablet_manager.cc:1367] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.010s
I20240503 08:12:30.533092 5079 raft_consensus.cc:348] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:30.533397 5079 raft_consensus.cc:374] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.533495 5079 raft_consensus.cc:729] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.533887 5079 consensus_queue.cc:260] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:30.539537 5097 raft_consensus.cc:482] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:30.539853 5097 raft_consensus.cc:504] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:30.542105 5097 leader_election.cc:288] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:30.544557 5079 ts_tablet_manager.cc:1398] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240503 08:12:30.554018 5112 tablet_bootstrap.cc:492] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:30.554219 5112 ts_tablet_manager.cc:1367] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.005s
I20240503 08:12:30.555194 5112 raft_consensus.cc:348] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:30.555406 5090 raft_consensus.cc:482] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:30.555696 5112 raft_consensus.cc:374] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.555858 5112 raft_consensus.cc:729] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.555658 5090 raft_consensus.cc:504] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:30.556201 5112 consensus_queue.cc:260] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:30.556864 5112 ts_tablet_manager.cc:1398] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:30.557758 5090 leader_election.cc:288] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:30.567783 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4d06fdfc4e49d5b0790e2ee3b4cc83" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:30.568225 4703 raft_consensus.cc:2454] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:30.568769 2010 leader_election.cc:302] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:30.569114 4134 raft_consensus.cc:2781] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:30.569233 4134 raft_consensus.cc:482] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:30.569342 4134 raft_consensus.cc:3037] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.572368 4134 raft_consensus.cc:504] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.573169 4134 leader_election.cc:288] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
W20240503 08:12:30.573994 5014 tablet.cc:2369] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:30.574347 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4d06fdfc4e49d5b0790e2ee3b4cc83" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:30.574626 4703 raft_consensus.cc:3037] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.584689 4703 raft_consensus.cc:2454] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:30.585261 2010 leader_election.cc:302] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:30.585620 4134 raft_consensus.cc:2781] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:30.586161 4134 raft_consensus.cc:686] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:30.586512 4134 consensus_queue.cc:237] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.601246 5113 tablet_bootstrap.cc:654] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:30.606251 1668 catalog_manager.cc:5484] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:30.607663 5113 tablet_bootstrap.cc:492] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:30.607968 5113 ts_tablet_manager.cc:1367] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.003s
I20240503 08:12:30.608934 5113 raft_consensus.cc:348] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:30.609334 5113 raft_consensus.cc:374] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:30.609483 5113 raft_consensus.cc:729] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:30.609815 5113 consensus_queue.cc:260] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:30.610656 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4d06fdfc4e49d5b0790e2ee3b4cc83" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:30.610965 4967 raft_consensus.cc:2454] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:30.613269 5113 ts_tablet_manager.cc:1398] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 08:12:30.614826 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8737ef0936c641ccaf0b804b1d9c3e07" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:30.615139 4703 raft_consensus.cc:2454] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:30.616606 4903 leader_election.cc:302] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:30.617169 5097 raft_consensus.cc:2781] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:30.617349 5097 raft_consensus.cc:482] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:30.617452 5097 raft_consensus.cc:3037] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.617383 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4d06fdfc4e49d5b0790e2ee3b4cc83" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:30.617607 4967 raft_consensus.cc:3037] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.620325 5055 raft_consensus.cc:482] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:30.620509 5055 raft_consensus.cc:504] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:30.620888 5097 raft_consensus.cc:504] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:30.621081 4967 raft_consensus.cc:2454] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:30.621505 5055 leader_election.cc:288] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:30.622014 5097 leader_election.cc:288] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:30.625231 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8737ef0936c641ccaf0b804b1d9c3e07" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:30.625604 4703 raft_consensus.cc:3037] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.625540 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8737ef0936c641ccaf0b804b1d9c3e07" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:30.625962 4967 raft_consensus.cc:2379] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31f41542452b4b1eb994ff4223d039fc in current term 1: Already voted for candidate 00b61625d8a7460892c118a2570afda5 in this term.
I20240503 08:12:30.627311 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8737ef0936c641ccaf0b804b1d9c3e07" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:30.627604 2203 raft_consensus.cc:2454] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:30.629813 4636 leader_election.cc:302] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters: 00b61625d8a7460892c118a2570afda5
I20240503 08:12:30.630040 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8737ef0936c641ccaf0b804b1d9c3e07" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:30.630131 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8737ef0936c641ccaf0b804b1d9c3e07" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:30.630321 2203 raft_consensus.cc:2454] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:30.636428 4703 raft_consensus.cc:2454] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:30.636629 5090 raft_consensus.cc:2741] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 08:12:30.637964 4903 leader_election.cc:302] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 31f41542452b4b1eb994ff4223d039fc; no voters: 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:30.638566 1671 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 08:12:30.638813 1671 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 08:12:30.641387 5097 raft_consensus.cc:2781] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:30.642076 5097 raft_consensus.cc:686] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:30.642792 5097 consensus_queue.cc:237] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:30.645233 1671 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 6f932635e2bf4c9da5f5f194975274a3
I20240503 08:12:30.646319 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f932635e2bf4c9da5f5f194975274a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:30.646659 5068 tablet_replica.cc:330] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:30.646888 1671 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 7071eecd95ca42159859ff358fcf9119
I20240503 08:12:30.646900 5068 raft_consensus.cc:2229] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:30.646950 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f932635e2bf4c9da5f5f194975274a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:30.647245 5067 tablet_replica.cc:330] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:30.647361 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f932635e2bf4c9da5f5f194975274a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:30.647477 5067 raft_consensus.cc:2229] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.647483 5068 raft_consensus.cc:2258] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.647645 5060 tablet_replica.cc:330] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:30.647780 5067 raft_consensus.cc:2258] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.647866 5060 raft_consensus.cc:2229] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.647881 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet 7071eecd95ca42159859ff358fcf9119 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:30.648138 5060 raft_consensus.cc:2258] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.648183 5066 tablet_replica.cc:330] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:30.648396 5066 raft_consensus.cc:2229] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.648698 5066 raft_consensus.cc:2258] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.649293 5060 ts_tablet_manager.cc:1875] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.649550 5066 ts_tablet_manager.cc:1875] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.651072 5067 ts_tablet_manager.cc:1875] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.648346 5068 ts_tablet_manager.cc:1875] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.657069 1671 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet a82d95d9203d41348c46fc95a4cd75fd
I20240503 08:12:30.661250 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 7071eecd95ca42159859ff358fcf9119 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:30.661559 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet a82d95d9203d41348c46fc95a4cd75fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:30.663465 5066 ts_tablet_manager.cc:1888] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.663620 5066 log.cc:1196] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/7071eecd95ca42159859ff358fcf9119
I20240503 08:12:30.664165 5066 ts_tablet_manager.cc:1909] T 7071eecd95ca42159859ff358fcf9119 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:30.665056 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet a82d95d9203d41348c46fc95a4cd75fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:30.666725 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 7071eecd95ca42159859ff358fcf9119 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:30.667194 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet a82d95d9203d41348c46fc95a4cd75fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:30 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:30.667562 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 7071eecd95ca42159859ff358fcf9119 (table rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]) successfully deleted
I20240503 08:12:30.668099 5067 ts_tablet_manager.cc:1888] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.668788 5060 ts_tablet_manager.cc:1888] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.668807 5067 log.cc:1196] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f932635e2bf4c9da5f5f194975274a3
I20240503 08:12:30.668920 5060 log.cc:1196] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/6f932635e2bf4c9da5f5f194975274a3
I20240503 08:12:30.669366 5060 ts_tablet_manager.cc:1909] T 6f932635e2bf4c9da5f5f194975274a3 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:30.669863 5068 ts_tablet_manager.cc:1888] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.670055 5068 log.cc:1196] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f932635e2bf4c9da5f5f194975274a3
I20240503 08:12:30.670571 5068 ts_tablet_manager.cc:1909] T 6f932635e2bf4c9da5f5f194975274a3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:30.672516 5060 tablet_replica.cc:330] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:30.672467 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 6f932635e2bf4c9da5f5f194975274a3 (table rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]) successfully deleted
I20240503 08:12:30.672535 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 6f932635e2bf4c9da5f5f194975274a3 (table rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]) successfully deleted
I20240503 08:12:30.673003 5060 raft_consensus.cc:2229] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.673179 5075 raft_consensus.cc:482] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:30.673288 5060 raft_consensus.cc:2258] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.673323 5075 raft_consensus.cc:504] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:30.674047 5060 ts_tablet_manager.cc:1875] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.674257 5075 leader_election.cc:288] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:30.674774 1671 catalog_manager.cc:5484] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:30.683140 5068 tablet_replica.cc:330] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:30.677281 5067 ts_tablet_manager.cc:1909] T 6f932635e2bf4c9da5f5f194975274a3 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:30.684741 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 6f932635e2bf4c9da5f5f194975274a3 (table rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]) successfully deleted
I20240503 08:12:30.686483 5068 raft_consensus.cc:2229] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.686869 5068 raft_consensus.cc:2258] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.687567 5067 tablet_replica.cc:330] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:30.687875 5067 raft_consensus.cc:2229] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:30.688407 5068 ts_tablet_manager.cc:1875] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.688658 5067 raft_consensus.cc:2258] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.691469 5067 ts_tablet_manager.cc:1875] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.695889 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d70c0406fe734040858907f3b68fdd3c" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:30.696166 4703 raft_consensus.cc:2454] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:30.696671 4504 leader_election.cc:302] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:30.697010 5055 raft_consensus.cc:2781] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:30.697134 5055 raft_consensus.cc:482] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:30.697230 5055 raft_consensus.cc:3037] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.700354 5055 raft_consensus.cc:504] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:30.701578 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d70c0406fe734040858907f3b68fdd3c" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:30.701798 4703 raft_consensus.cc:3037] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.701494 5055 leader_election.cc:288] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:30.702100 5067 ts_tablet_manager.cc:1888] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.702260 5067 log.cc:1196] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/7071eecd95ca42159859ff358fcf9119
I20240503 08:12:30.702730 5067 ts_tablet_manager.cc:1909] T 7071eecd95ca42159859ff358fcf9119 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:30.703892 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 7071eecd95ca42159859ff358fcf9119 (table rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]) successfully deleted
I20240503 08:12:30.704257 4703 raft_consensus.cc:2454] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:30.704957 4504 leader_election.cc:302] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:30.705006 5060 ts_tablet_manager.cc:1888] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.705159 5060 log.cc:1196] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/7071eecd95ca42159859ff358fcf9119
I20240503 08:12:30.705485 5055 raft_consensus.cc:2781] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:30.705603 5060 ts_tablet_manager.cc:1909] T 7071eecd95ca42159859ff358fcf9119 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:30.705659 5055 raft_consensus.cc:686] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:30.706235 5055 consensus_queue.cc:237] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:30.706825 5060 tablet_replica.cc:330] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:30.706949 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 7071eecd95ca42159859ff358fcf9119 (table rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]) successfully deleted
I20240503 08:12:30.707098 5060 raft_consensus.cc:2229] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:30.707481 5060 raft_consensus.cc:2258] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.708384 5060 ts_tablet_manager.cc:1875] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.711479 5067 tablet_replica.cc:330] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:30.711814 5067 raft_consensus.cc:2229] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:30.712916 5067 raft_consensus.cc:2258] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:30.713729 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d70c0406fe734040858907f3b68fdd3c" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:30.713966 1943 raft_consensus.cc:3037] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.714010 5067 ts_tablet_manager.cc:1875] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:30.715155 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d70c0406fe734040858907f3b68fdd3c" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:30.718353 1943 raft_consensus.cc:2454] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:30.730114 5068 ts_tablet_manager.cc:1888] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.732337 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d70c0406fe734040858907f3b68fdd3c" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:30.732707 4703 raft_consensus.cc:2379] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in current term 1: Already voted for candidate 5f95b05d111e4090bbc59dec394ed0ab in this term.
I20240503 08:12:30.734747 1668 catalog_manager.cc:5484] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:30.737293 5068 log.cc:1196] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/a82d95d9203d41348c46fc95a4cd75fd
I20240503 08:12:30.738098 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d70c0406fe734040858907f3b68fdd3c" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:30.738891 1877 leader_election.cc:302] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:30.739568 4475 raft_consensus.cc:2726] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:30.739655 5068 ts_tablet_manager.cc:1909] T a82d95d9203d41348c46fc95a4cd75fd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:30.740808 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet a82d95d9203d41348c46fc95a4cd75fd (table rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]) successfully deleted
I20240503 08:12:30.741727 5067 ts_tablet_manager.cc:1888] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.741909 5067 log.cc:1196] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/a82d95d9203d41348c46fc95a4cd75fd
I20240503 08:12:30.742491 5067 ts_tablet_manager.cc:1909] T a82d95d9203d41348c46fc95a4cd75fd P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:30.743973 5060 ts_tablet_manager.cc:1888] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:30.744165 5060 log.cc:1196] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/a82d95d9203d41348c46fc95a4cd75fd
I20240503 08:12:30.745306 5060 ts_tablet_manager.cc:1909] T a82d95d9203d41348c46fc95a4cd75fd P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:30.746601 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet a82d95d9203d41348c46fc95a4cd75fd (table rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]) successfully deleted
I20240503 08:12:30.754828 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet a82d95d9203d41348c46fc95a4cd75fd (table rebalance_test_table_2 [id=d9fbd7a17c834dd8ad4e8a27476f51ad]) successfully deleted
I20240503 08:12:30.797753 5097 raft_consensus.cc:482] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:30.797986 5097 raft_consensus.cc:504] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:30.798699 5097 leader_election.cc:288] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:30.799618 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820ecfef09a34327a08f7c3a893ccb05" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:30.799927 2203 raft_consensus.cc:2379] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00b61625d8a7460892c118a2570afda5 in current term 1: Already voted for candidate f24648c92c384f1ba349eee801f985e2 in this term.
I20240503 08:12:30.829619 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820ecfef09a34327a08f7c3a893ccb05" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:30.830358 4903 leader_election.cc:302] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 184c543cb05d49cc98ba1c56617be7a5, f24648c92c384f1ba349eee801f985e2
I20240503 08:12:30.830754 5097 raft_consensus.cc:3037] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:30.833346 5097 raft_consensus.cc:2726] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:30.874181 1668 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:30.903596 5093 consensus_queue.cc:1035] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:30.905489 1943 raft_consensus.cc:1261] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:30.906385 5037 consensus_queue.cc:1035] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:30.908175 4703 raft_consensus.cc:1261] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:30.909150 5037 consensus_queue.cc:1035] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:30.965826 1668 catalog_manager.cc:5576] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975) does not have the latest schema for tablet d70c0406fe734040858907f3b68fdd3c (table rebalancer_dynamic_table [id=65d93adde78b4eb098ee656448a6ab57]). Expected version 1 got 0
I20240503 08:12:30.980715 5045 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:30.988241 5192 tablet.cc:1722] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:30.988996 5192 tablet.cc:1620] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:30.996414 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=65d93adde78b4eb098ee656448a6ab57] alter complete (version 1)
I20240503 08:12:30.997843 5193 tablet.cc:1722] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:31.000485 5193 tablet.cc:1620] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: MemRowSet was empty: no flush needed.
I20240503 08:12:31.008504 1671 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet d70c0406fe734040858907f3b68fdd3c (table rebalancer_dynamic_table [id=65d93adde78b4eb098ee656448a6ab57]). Expected version 1 got 0
I20240503 08:12:31.011111 1668 catalog_manager.cc:5576] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975) does not have the latest schema for tablet d70c0406fe734040858907f3b68fdd3c (table rebalancer_dynamic_table [id=65d93adde78b4eb098ee656448a6ab57]). Expected version 1 got 0
I20240503 08:12:31.020510 5190 consensus_queue.cc:1035] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:31.051723 5200 tablet.cc:1722] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:31.053761 1671 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet d70c0406fe734040858907f3b68fdd3c (table rebalancer_dynamic_table [id=65d93adde78b4eb098ee656448a6ab57]). Expected version 1 got 0
I20240503 08:12:31.056107 5200 tablet.cc:1620] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:31.062433 5045 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:31.062968 5097 consensus_queue.cc:1035] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:31.068207 2203 raft_consensus.cc:3037] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.079972 5097 consensus_queue.cc:1035] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:31.122378 1671 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:31.133677 5192 tablet.cc:1722] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:31.134284 5192 tablet.cc:1620] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:31.134581 5193 tablet.cc:1722] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:31.135062 5193 tablet.cc:1620] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: MemRowSet was empty: no flush needed.
I20240503 08:12:31.135382 5200 tablet.cc:1722] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:31.135839 5200 tablet.cc:1620] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:31.147958 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=65d93adde78b4eb098ee656448a6ab57] alter complete (version 2)
I20240503 08:12:31.163019 1665 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 08:12:31.163451 1665 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 08:12:31.169847 1665 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 23e53aef15cc4b80b1793a6a10eb5b2b
I20240503 08:12:31.170750 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e53aef15cc4b80b1793a6a10eb5b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:31.171042 5060 tablet_replica.cc:330] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:31.171279 5060 raft_consensus.cc:2229] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.171819 5060 raft_consensus.cc:2258] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.173485 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e53aef15cc4b80b1793a6a10eb5b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:31.173754 5060 ts_tablet_manager.cc:1875] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.173830 5067 tablet_replica.cc:330] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:31.174075 5067 raft_consensus.cc:2229] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.174340 5067 raft_consensus.cc:2258] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.175184 5067 ts_tablet_manager.cc:1875] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.176339 1665 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet f5408b903cd14f3384a68a12fe4abcbb
I20240503 08:12:31.176555 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 23e53aef15cc4b80b1793a6a10eb5b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:31.176848 5068 tablet_replica.cc:330] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:31.177103 5068 raft_consensus.cc:2229] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:31.177078 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet f5408b903cd14f3384a68a12fe4abcbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:31.177479 5068 raft_consensus.cc:2258] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.178452 5068 ts_tablet_manager.cc:1875] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.179183 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet f5408b903cd14f3384a68a12fe4abcbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:31.181219 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet f5408b903cd14f3384a68a12fe4abcbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:31.181815 5067 ts_tablet_manager.cc:1888] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.181975 5067 log.cc:1196] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/23e53aef15cc4b80b1793a6a10eb5b2b
I20240503 08:12:31.182444 5067 ts_tablet_manager.cc:1909] T 23e53aef15cc4b80b1793a6a10eb5b2b P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:31.183676 5067 tablet_replica.cc:330] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:31.183969 5067 raft_consensus.cc:2229] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.184288 5067 raft_consensus.cc:2258] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.184216 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 23e53aef15cc4b80b1793a6a10eb5b2b (table rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]) successfully deleted
I20240503 08:12:31.181632 1665 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 7890ae8e9d7d4b4ba072de7969c20fb0
I20240503 08:12:31.185305 5067 ts_tablet_manager.cc:1875] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.185329 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet 7890ae8e9d7d4b4ba072de7969c20fb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:35210
W20240503 08:12:31.186825 1879 consensus_peers.cc:485] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:31.187840 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 7890ae8e9d7d4b4ba072de7969c20fb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:31.188704 4947 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:31.190753 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet 7890ae8e9d7d4b4ba072de7969c20fb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:31.198905 5068 ts_tablet_manager.cc:1888] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.199554 5067 ts_tablet_manager.cc:1888] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.199740 5067 log.cc:1196] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/f5408b903cd14f3384a68a12fe4abcbb
I20240503 08:12:31.199872 5068 log.cc:1196] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/23e53aef15cc4b80b1793a6a10eb5b2b
I20240503 08:12:31.200232 5067 ts_tablet_manager.cc:1909] T f5408b903cd14f3384a68a12fe4abcbb P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:31.201426 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet f5408b903cd14f3384a68a12fe4abcbb (table rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]) successfully deleted
I20240503 08:12:31.201831 5068 ts_tablet_manager.cc:1909] T 23e53aef15cc4b80b1793a6a10eb5b2b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:31.203182 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 23e53aef15cc4b80b1793a6a10eb5b2b (table rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]) successfully deleted
I20240503 08:12:31.204988 5067 tablet_replica.cc:330] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:31.205250 5067 raft_consensus.cc:2229] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.205515 5067 raft_consensus.cc:2258] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.206310 5067 ts_tablet_manager.cc:1875] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.210454 5060 ts_tablet_manager.cc:1888] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.210640 5060 log.cc:1196] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/23e53aef15cc4b80b1793a6a10eb5b2b
I20240503 08:12:31.210929 5068 tablet_replica.cc:330] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:31.211537 5060 ts_tablet_manager.cc:1909] T 23e53aef15cc4b80b1793a6a10eb5b2b P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:31.211896 5068 raft_consensus.cc:2229] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:31.212848 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 23e53aef15cc4b80b1793a6a10eb5b2b (table rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]) successfully deleted
I20240503 08:12:31.215591 5068 raft_consensus.cc:2258] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.216055 5067 ts_tablet_manager.cc:1888] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.219041 5060 tablet_replica.cc:330] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:31.219276 5068 ts_tablet_manager.cc:1875] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.219406 5060 raft_consensus.cc:2229] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:31.220065 5060 raft_consensus.cc:2258] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.221575 5060 ts_tablet_manager.cc:1875] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.222064 5213 tablet_replica.cc:330] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:31.216212 5067 log.cc:1196] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/7890ae8e9d7d4b4ba072de7969c20fb0
I20240503 08:12:31.222829 5067 ts_tablet_manager.cc:1909] T 7890ae8e9d7d4b4ba072de7969c20fb0 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:31.223140 5213 raft_consensus.cc:2229] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.223995 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 7890ae8e9d7d4b4ba072de7969c20fb0 (table rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]) successfully deleted
I20240503 08:12:31.224486 5213 raft_consensus.cc:2258] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.228132 5213 ts_tablet_manager.cc:1875] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.235786 2184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:12:31.251717 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:31.261720 5192 tablet.cc:1722] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:31.262756 5068 ts_tablet_manager.cc:1888] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.263454 5068 log.cc:1196] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/f5408b903cd14f3384a68a12fe4abcbb
I20240503 08:12:31.267910 5068 ts_tablet_manager.cc:1909] T f5408b903cd14f3384a68a12fe4abcbb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:31.268229 5200 tablet.cc:1722] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:31.269181 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet f5408b903cd14f3384a68a12fe4abcbb (table rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]) successfully deleted
I20240503 08:12:31.272110 1666 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:31.273581 1670 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=65d93adde78b4eb098ee656448a6ab57] alter complete (version 3)
I20240503 08:12:31.285178 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 34b7fd8dc7434b8ea14b9deb0023b4cb (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.285687 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b7fd8dc7434b8ea14b9deb0023b4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.286222 4947 tablet_service.cc:1467] Processing CreateTablet for tablet 34b7fd8dc7434b8ea14b9deb0023b4cb (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.286711 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b7fd8dc7434b8ea14b9deb0023b4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.289799 4549 tablet_service.cc:1467] Processing CreateTablet for tablet 34b7fd8dc7434b8ea14b9deb0023b4cb (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.290426 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b7fd8dc7434b8ea14b9deb0023b4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.291750 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4f6c5975f8446a9f6ba8cae137bda0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.292210 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4f6c5975f8446a9f6ba8cae137bda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.292205 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 79579b65538d47f7973fa51f973b3820 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.292215 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4f6c5975f8446a9f6ba8cae137bda0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.292623 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4f6c5975f8446a9f6ba8cae137bda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.292640 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79579b65538d47f7973fa51f973b3820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.295831 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 79579b65538d47f7973fa51f973b3820 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.296296 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79579b65538d47f7973fa51f973b3820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.296537 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4f6c5975f8446a9f6ba8cae137bda0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.297093 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4f6c5975f8446a9f6ba8cae137bda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.300980 5193 tablet.cc:1722] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:31.302567 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 79579b65538d47f7973fa51f973b3820 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:31.303074 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79579b65538d47f7973fa51f973b3820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.319226 5222 tablet_bootstrap.cc:492] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:31.322512 5222 tablet_bootstrap.cc:654] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.333123 5222 tablet_bootstrap.cc:492] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:31.333465 5222 ts_tablet_manager.cc:1367] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20240503 08:12:31.334038 5223 tablet_bootstrap.cc:492] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:31.334560 5222 raft_consensus.cc:348] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.334945 5222 raft_consensus.cc:374] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.335098 5222 raft_consensus.cc:729] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.335484 5222 consensus_queue.cc:260] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.336213 5223 tablet_bootstrap.cc:654] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.337628 5225 tablet_bootstrap.cc:492] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:31.339833 5224 tablet_bootstrap.cc:492] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:31.340063 5222 ts_tablet_manager.cc:1398] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:31.342332 5224 tablet_bootstrap.cc:654] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.343138 5222 tablet_bootstrap.cc:492] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:31.345189 5222 tablet_bootstrap.cc:654] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.346149 5231 tablet_bootstrap.cc:492] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:31.349622 5231 tablet_bootstrap.cc:654] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.352038 5213 ts_tablet_manager.cc:1888] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.352298 5213 log.cc:1196] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/f5408b903cd14f3384a68a12fe4abcbb
I20240503 08:12:31.353848 5213 ts_tablet_manager.cc:1909] T f5408b903cd14f3384a68a12fe4abcbb P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:31.355085 5213 tablet_replica.cc:330] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:31.355430 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet f5408b903cd14f3384a68a12fe4abcbb (table rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]) successfully deleted
I20240503 08:12:31.355475 5213 raft_consensus.cc:2229] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.355722 5213 raft_consensus.cc:2258] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.356062 5231 tablet_bootstrap.cc:492] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:31.347290 5060 ts_tablet_manager.cc:1888] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.363801 5060 log.cc:1196] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/7890ae8e9d7d4b4ba072de7969c20fb0
I20240503 08:12:31.364306 5060 ts_tablet_manager.cc:1909] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:31.365434 5231 ts_tablet_manager.cc:1367] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.006s
I20240503 08:12:31.365460 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 7890ae8e9d7d4b4ba072de7969c20fb0 (table rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]) successfully deleted
I20240503 08:12:31.365625 1924 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:12:31.367874 5231 raft_consensus.cc:348] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:31.369300 5222 tablet_bootstrap.cc:492] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:31.369457 5225 tablet_bootstrap.cc:654] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.369526 5222 ts_tablet_manager.cc:1367] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.003s
I20240503 08:12:31.370353 5222 raft_consensus.cc:348] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.370648 5222 raft_consensus.cc:374] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.370756 5222 raft_consensus.cc:729] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.371078 5222 consensus_queue.cc:260] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.372607 5223 tablet_bootstrap.cc:492] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:31.372829 5223 ts_tablet_manager.cc:1367] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240503 08:12:31.374926 5222 ts_tablet_manager.cc:1398] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:12:31.378612 1666 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:31.379014 1666 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:31.379170 5225 tablet_bootstrap.cc:492] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:31.379410 5225 ts_tablet_manager.cc:1367] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240503 08:12:31.383806 1666 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet d70c0406fe734040858907f3b68fdd3c
I20240503 08:12:31.385004 5225 raft_consensus.cc:348] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.386454 5233 tablet_bootstrap.cc:492] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:31.386847 5224 tablet_bootstrap.cc:492] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:31.387697 5224 ts_tablet_manager.cc:1367] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20240503 08:12:31.387974 5234 tablet_bootstrap.cc:492] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:31.390249 5234 tablet_bootstrap.cc:654] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.390647 5224 raft_consensus.cc:348] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.390892 5224 raft_consensus.cc:374] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.391000 5224 raft_consensus.cc:729] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.391238 5224 consensus_queue.cc:260] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.375545 5223 raft_consensus.cc:348] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:31.392352 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet d70c0406fe734040858907f3b68fdd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:31.392582 5223 raft_consensus.cc:374] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.392689 5223 raft_consensus.cc:729] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.393769 5233 tablet_bootstrap.cc:654] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.393728 5223 consensus_queue.cc:260] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:31.394311 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet d70c0406fe734040858907f3b68fdd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:31.396359 5068 tablet_replica.cc:330] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:31.396615 5068 raft_consensus.cc:2229] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.396835 5068 raft_consensus.cc:2258] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.397428 5223 ts_tablet_manager.cc:1398] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240503 08:12:31.394317 5225 raft_consensus.cc:374] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.397728 5225 raft_consensus.cc:729] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.397987 5225 consensus_queue.cc:260] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.398674 5225 ts_tablet_manager.cc:1398] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 08:12:31.398993 5225 tablet_bootstrap.cc:492] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:31.399775 5068 ts_tablet_manager.cc:1875] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.400053 5224 ts_tablet_manager.cc:1398] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:12:31.402940 5225 tablet_bootstrap.cc:654] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.403172 5242 tablet_replica.cc:330] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:31.403733 5242 raft_consensus.cc:2229] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.404250 5242 raft_consensus.cc:2258] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.400285 5213 ts_tablet_manager.cc:1875] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.416847 5197 raft_consensus.cc:482] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:31.417104 5197 raft_consensus.cc:504] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.418431 5197 leader_election.cc:288] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:31.421506 5068 ts_tablet_manager.cc:1888] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:31.421921 5068 log.cc:1196] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/d70c0406fe734040858907f3b68fdd3c
I20240503 08:12:31.422559 5068 ts_tablet_manager.cc:1909] T d70c0406fe734040858907f3b68fdd3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:31.391628 4549 tablet_service.cc:1514] Processing DeleteTablet for tablet d70c0406fe734040858907f3b68fdd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:31.424386 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet d70c0406fe734040858907f3b68fdd3c (table rebalancer_dynamic_table_new_name [id=65d93adde78b4eb098ee656448a6ab57]) successfully deleted
I20240503 08:12:31.422848 5231 raft_consensus.cc:374] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.425105 5231 raft_consensus.cc:729] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.425385 5231 consensus_queue.cc:260] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:31.426033 5231 ts_tablet_manager.cc:1398] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.060s user 0.001s sys 0.000s
I20240503 08:12:31.427510 5248 tablet_replica.cc:330] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:31.410006 4682 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:31.429481 5248 raft_consensus.cc:2229] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:31.430025 5248 raft_consensus.cc:2258] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.431694 5242 ts_tablet_manager.cc:1875] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.433178 5248 ts_tablet_manager.cc:1875] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.452947 5248 ts_tablet_manager.cc:1888] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:31.453109 2053 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 08:12:31.455860 5234 tablet_bootstrap.cc:492] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:31.456135 5234 ts_tablet_manager.cc:1367] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.001s
I20240503 08:12:31.456936 5248 log.cc:1196] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/d70c0406fe734040858907f3b68fdd3c
I20240503 08:12:31.457506 5248 ts_tablet_manager.cc:1909] T d70c0406fe734040858907f3b68fdd3c P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:31.460327 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet d70c0406fe734040858907f3b68fdd3c (table rebalancer_dynamic_table_new_name [id=65d93adde78b4eb098ee656448a6ab57]) successfully deleted
I20240503 08:12:31.460565 5225 tablet_bootstrap.cc:492] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:31.460896 5225 ts_tablet_manager.cc:1367] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.062s user 0.016s sys 0.023s
I20240503 08:12:31.461989 5225 raft_consensus.cc:348] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:31.462320 5225 raft_consensus.cc:374] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.462436 5225 raft_consensus.cc:729] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.462793 5225 consensus_queue.cc:260] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:31.464656 4549 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:31.470414 5225 ts_tablet_manager.cc:1398] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240503 08:12:31.470933 5233 tablet_bootstrap.cc:492] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:31.471172 5233 ts_tablet_manager.cc:1367] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.000s
I20240503 08:12:31.472143 5233 raft_consensus.cc:348] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.472379 5233 raft_consensus.cc:374] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.475507 5234 raft_consensus.cc:348] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.475792 5234 raft_consensus.cc:374] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.475914 5234 raft_consensus.cc:729] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.476197 5234 consensus_queue.cc:260] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.477234 5234 ts_tablet_manager.cc:1398] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240503 08:12:31.494706 5213 ts_tablet_manager.cc:1888] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.494899 5213 log.cc:1196] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/7890ae8e9d7d4b4ba072de7969c20fb0
I20240503 08:12:31.495471 5213 ts_tablet_manager.cc:1909] T 7890ae8e9d7d4b4ba072de7969c20fb0 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:31.496012 5233 raft_consensus.cc:729] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.499058 5233 consensus_queue.cc:260] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.503418 5233 ts_tablet_manager.cc:1398] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.032s user 0.000s sys 0.003s
I20240503 08:12:31.504480 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4f6c5975f8446a9f6ba8cae137bda0" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:31.504734 1944 raft_consensus.cc:2454] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:31.505295 4769 leader_election.cc:302] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:31.505684 5197 raft_consensus.cc:2781] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:31.505797 5197 raft_consensus.cc:482] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:31.505873 5197 raft_consensus.cc:3037] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.505813 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:31.510720 4814 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:31.515381 5197 raft_consensus.cc:504] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.517966 5197 leader_election.cc:288] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:31.518069 5045 raft_consensus.cc:482] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:31.518214 5045 raft_consensus.cc:504] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.519002 5045 leader_election.cc:288] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:31.521947 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4f6c5975f8446a9f6ba8cae137bda0" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:31.522202 1944 raft_consensus.cc:3037] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.523954 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79579b65538d47f7973fa51f973b3820" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:31.524212 4703 raft_consensus.cc:2454] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:31.525344 1944 raft_consensus.cc:2454] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:31.525559 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79579b65538d47f7973fa51f973b3820" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:31.525820 1811 raft_consensus.cc:2454] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:31.526329 2009 leader_election.cc:302] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:31.527045 5045 raft_consensus.cc:2781] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:31.527169 5045 raft_consensus.cc:482] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:31.527247 5045 raft_consensus.cc:3037] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.529983 4682 tablet_service.cc:1467] Processing CreateTablet for tablet e2da49afae5843b5a45e9ebaa67f3d0c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f08a2224751b42ecaee59ed46747f235]), partition=
I20240503 08:12:31.530120 5045 raft_consensus.cc:504] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.530392 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2da49afae5843b5a45e9ebaa67f3d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.532092 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 7890ae8e9d7d4b4ba072de7969c20fb0 (table rebalance_test_table_4 [id=95ff899f1ac9441b92ca68e7bdb67776]) successfully deleted
I20240503 08:12:31.532475 4549 tablet_service.cc:1467] Processing CreateTablet for tablet e2da49afae5843b5a45e9ebaa67f3d0c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f08a2224751b42ecaee59ed46747f235]), partition=
I20240503 08:12:31.532925 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2da49afae5843b5a45e9ebaa67f3d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.534076 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79579b65538d47f7973fa51f973b3820" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:31.534327 4702 raft_consensus.cc:3037] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.537199 5090 raft_consensus.cc:482] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:31.537367 5090 raft_consensus.cc:504] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.538254 5090 leader_election.cc:288] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:31.537354 4702 raft_consensus.cc:2454] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:31.541002 5231 tablet_bootstrap.cc:492] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:31.542301 4769 leader_election.cc:302] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:31.542651 5231 tablet_bootstrap.cc:654] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.543036 5190 raft_consensus.cc:2781] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:31.543232 5190 raft_consensus.cc:686] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:31.543726 5190 consensus_queue.cc:237] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.547497 2010 leader_election.cc:302] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:31.548171 4885 raft_consensus.cc:2781] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:31.548218 1792 tablet_service.cc:1467] Processing CreateTablet for tablet e2da49afae5843b5a45e9ebaa67f3d0c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f08a2224751b42ecaee59ed46747f235]), partition=
I20240503 08:12:31.548624 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2da49afae5843b5a45e9ebaa67f3d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:31.549420 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79579b65538d47f7973fa51f973b3820" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:31.549650 1811 raft_consensus.cc:3037] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.552693 1811 raft_consensus.cc:2454] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:31.556178 5045 leader_election.cc:288] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:31.557397 5222 tablet_bootstrap.cc:492] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:31.559481 5222 tablet_bootstrap.cc:654] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.562050 5222 tablet_bootstrap.cc:492] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:31.562283 5222 ts_tablet_manager.cc:1367] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 08:12:31.563272 5222 raft_consensus.cc:348] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:31.563532 5222 raft_consensus.cc:374] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.563650 5222 raft_consensus.cc:729] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.563891 5222 consensus_queue.cc:260] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:31.565456 5222 ts_tablet_manager.cc:1398] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:31.565945 5233 tablet_bootstrap.cc:492] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:31.571935 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4f6c5975f8446a9f6ba8cae137bda0" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:31.572242 4703 raft_consensus.cc:2454] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:31.572873 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4f6c5975f8446a9f6ba8cae137bda0" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:31.573089 4702 raft_consensus.cc:3037] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.574009 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4f6c5975f8446a9f6ba8cae137bda0" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:31.577461 4702 raft_consensus.cc:2454] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:31.587077 1665 catalog_manager.cc:5484] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:31.587981 5233 tablet_bootstrap.cc:654] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:31.590312 5242 ts_tablet_manager.cc:1888] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:31.596269 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4f6c5975f8446a9f6ba8cae137bda0" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:31.596630 1944 raft_consensus.cc:2379] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31f41542452b4b1eb994ff4223d039fc in current term 1: Already voted for candidate f24648c92c384f1ba349eee801f985e2 in this term.
I20240503 08:12:31.606833 4885 raft_consensus.cc:686] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:31.607478 4885 consensus_queue.cc:237] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:31.609957 1792 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:12:31.597292 4637 leader_election.cc:302] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, f24648c92c384f1ba349eee801f985e2
I20240503 08:12:31.616026 5090 raft_consensus.cc:2726] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:31.616750 5242 log.cc:1196] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/d70c0406fe734040858907f3b68fdd3c
I20240503 08:12:31.618501 5231 tablet_bootstrap.cc:492] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:31.618780 5231 ts_tablet_manager.cc:1367] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.078s user 0.005s sys 0.001s
I20240503 08:12:31.619781 5231 raft_consensus.cc:348] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:31.620045 5231 raft_consensus.cc:374] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.620517 5231 raft_consensus.cc:729] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.620925 5231 consensus_queue.cc:260] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:31.621250 5233 tablet_bootstrap.cc:492] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:31.621495 5233 ts_tablet_manager.cc:1367] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.002s
I20240503 08:12:31.621731 5231 ts_tablet_manager.cc:1398] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:31.622465 5233 raft_consensus.cc:348] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:31.622786 5233 raft_consensus.cc:374] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:31.622905 5233 raft_consensus.cc:729] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:31.623299 5233 consensus_queue.cc:260] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:31.624002 5233 ts_tablet_manager.cc:1398] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:31.625061 5242 ts_tablet_manager.cc:1909] T d70c0406fe734040858907f3b68fdd3c P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:31.628902 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet d70c0406fe734040858907f3b68fdd3c (table rebalancer_dynamic_table_new_name [id=65d93adde78b4eb098ee656448a6ab57]) successfully deleted
I20240503 08:12:31.661504 1666 catalog_manager.cc:5484] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:31.670830 1859 tablet.cc:2369] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:31.692814 1666 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 08:12:31.693094 1666 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 08:12:31.698215 1666 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet f5fc5f680f1e43cf8083a558b14e4461
I20240503 08:12:31.698858 1666 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet a04d01412b4f420884570e24c9b204b8
I20240503 08:12:31.699261 1666 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 657d9622b1a241a0bed20e3474a0093a
I20240503 08:12:31.700740 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet a04d01412b4f420884570e24c9b204b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:31.700783 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet f5fc5f680f1e43cf8083a558b14e4461 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:31.701089 5060 tablet_replica.cc:330] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:31.701340 5060 raft_consensus.cc:2229] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.701581 5060 raft_consensus.cc:2258] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.702327 5060 ts_tablet_manager.cc:1875] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.703495 5067 tablet_replica.cc:330] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:31.703720 5067 raft_consensus.cc:2229] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.703982 5067 raft_consensus.cc:2258] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.704560 5067 ts_tablet_manager.cc:1875] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.707998 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet f5fc5f680f1e43cf8083a558b14e4461 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:31.708256 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet f5fc5f680f1e43cf8083a558b14e4461 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:31.708339 5213 tablet_replica.cc:330] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:31.708420 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet a04d01412b4f420884570e24c9b204b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:31.708554 5213 raft_consensus.cc:2229] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:31.708665 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 657d9622b1a241a0bed20e3474a0093a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:31.708938 5213 raft_consensus.cc:2258] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.708930 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 657d9622b1a241a0bed20e3474a0093a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:31.709100 5068 tablet_replica.cc:330] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:31.709134 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 657d9622b1a241a0bed20e3474a0093a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:31.709316 5068 raft_consensus.cc:2229] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.709553 5068 raft_consensus.cc:2258] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.709725 5213 ts_tablet_manager.cc:1875] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.710264 5068 ts_tablet_manager.cc:1875] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.714038 5067 ts_tablet_manager.cc:1888] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.714205 5067 log.cc:1196] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/f5fc5f680f1e43cf8083a558b14e4461
I20240503 08:12:31.714690 5067 ts_tablet_manager.cc:1909] T f5fc5f680f1e43cf8083a558b14e4461 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:31.715502 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet a04d01412b4f420884570e24c9b204b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:31 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:31.715751 5067 tablet_replica.cc:330] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:31.715988 5067 raft_consensus.cc:2229] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.716256 5067 raft_consensus.cc:2258] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.716228 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet f5fc5f680f1e43cf8083a558b14e4461 (table rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]) successfully deleted
I20240503 08:12:31.717008 5067 ts_tablet_manager.cc:1875] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.720616 5213 ts_tablet_manager.cc:1888] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.720835 5213 log.cc:1196] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/f5fc5f680f1e43cf8083a558b14e4461
I20240503 08:12:31.721338 5213 ts_tablet_manager.cc:1909] T f5fc5f680f1e43cf8083a558b14e4461 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:31.722618 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet f5fc5f680f1e43cf8083a558b14e4461 (table rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]) successfully deleted
I20240503 08:12:31.723029 5213 tablet_replica.cc:330] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:31.723338 5213 raft_consensus.cc:2229] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:31.723742 5213 raft_consensus.cc:2258] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.724540 5213 ts_tablet_manager.cc:1875] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.727115 5068 ts_tablet_manager.cc:1888] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.727291 5068 log.cc:1196] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/f5fc5f680f1e43cf8083a558b14e4461
I20240503 08:12:31.727761 5068 ts_tablet_manager.cc:1909] T f5fc5f680f1e43cf8083a558b14e4461 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:31.728906 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet f5fc5f680f1e43cf8083a558b14e4461 (table rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]) successfully deleted
I20240503 08:12:31.729614 5060 ts_tablet_manager.cc:1888] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.729782 5060 log.cc:1196] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/a04d01412b4f420884570e24c9b204b8
I20240503 08:12:31.730197 5060 ts_tablet_manager.cc:1909] T a04d01412b4f420884570e24c9b204b8 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:31.731278 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet a04d01412b4f420884570e24c9b204b8 (table rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]) successfully deleted
I20240503 08:12:31.732786 5213 ts_tablet_manager.cc:1888] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.733013 5213 log.cc:1196] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/a04d01412b4f420884570e24c9b204b8
I20240503 08:12:31.733410 5068 tablet_replica.cc:330] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:31.733609 5213 ts_tablet_manager.cc:1909] T a04d01412b4f420884570e24c9b204b8 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:31.733636 5068 raft_consensus.cc:2229] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:31.734009 5068 raft_consensus.cc:2258] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.734884 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet a04d01412b4f420884570e24c9b204b8 (table rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]) successfully deleted
I20240503 08:12:31.735311 5213 tablet_replica.cc:330] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:31.735579 5213 raft_consensus.cc:2229] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.735868 5068 ts_tablet_manager.cc:1875] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.735878 5213 raft_consensus.cc:2258] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.736680 5213 ts_tablet_manager.cc:1875] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.739513 5067 ts_tablet_manager.cc:1888] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.739724 5067 log.cc:1196] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/657d9622b1a241a0bed20e3474a0093a
I20240503 08:12:31.741027 5067 ts_tablet_manager.cc:1909] T 657d9622b1a241a0bed20e3474a0093a P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:31.742163 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 657d9622b1a241a0bed20e3474a0093a (table rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]) successfully deleted
I20240503 08:12:31.744138 5213 ts_tablet_manager.cc:1888] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.744740 5213 log.cc:1196] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/657d9622b1a241a0bed20e3474a0093a
I20240503 08:12:31.744973 5068 ts_tablet_manager.cc:1888] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.745131 5068 log.cc:1196] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/657d9622b1a241a0bed20e3474a0093a
I20240503 08:12:31.745398 5213 ts_tablet_manager.cc:1909] T 657d9622b1a241a0bed20e3474a0093a P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:31.745700 5068 ts_tablet_manager.cc:1909] T 657d9622b1a241a0bed20e3474a0093a P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:31.746637 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 657d9622b1a241a0bed20e3474a0093a (table rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]) successfully deleted
I20240503 08:12:31.746755 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 657d9622b1a241a0bed20e3474a0093a (table rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]) successfully deleted
I20240503 08:12:31.747581 5067 tablet_replica.cc:330] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:31.747822 5067 raft_consensus.cc:2229] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:31.748099 5067 raft_consensus.cc:2258] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:31.748831 5067 ts_tablet_manager.cc:1875] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:31.756788 5067 ts_tablet_manager.cc:1888] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:31.756945 5067 log.cc:1196] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/a04d01412b4f420884570e24c9b204b8
I20240503 08:12:31.757269 5090 raft_consensus.cc:482] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:31.757390 5067 ts_tablet_manager.cc:1909] T a04d01412b4f420884570e24c9b204b8 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:31.757450 5090 raft_consensus.cc:504] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:31.758431 5090 leader_election.cc:288] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:31.759042 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet a04d01412b4f420884570e24c9b204b8 (table rebalance_test_table_6 [id=bf4cdb3191df4dc1a55e396009fc991b]) successfully deleted
I20240503 08:12:31.767971 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2da49afae5843b5a45e9ebaa67f3d0c" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:31.768277 4569 raft_consensus.cc:2454] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:31.769181 4635 leader_election.cc:302] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:31.769991 5090 raft_consensus.cc:2781] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:31.770117 5090 raft_consensus.cc:482] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:31.770212 5090 raft_consensus.cc:3037] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.772430 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2da49afae5843b5a45e9ebaa67f3d0c" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:31.772683 1811 raft_consensus.cc:2454] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:31.773800 5090 raft_consensus.cc:504] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:31.774705 5090 leader_election.cc:288] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:31.775173 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2da49afae5843b5a45e9ebaa67f3d0c" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:31.775418 4569 raft_consensus.cc:3037] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.775465 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2da49afae5843b5a45e9ebaa67f3d0c" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:31.775656 1811 raft_consensus.cc:3037] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.778379 4569 raft_consensus.cc:2454] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:31.778426 1811 raft_consensus.cc:2454] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:31.779119 4635 leader_election.cc:302] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:31.779729 5090 raft_consensus.cc:2781] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:31.780288 5090 raft_consensus.cc:686] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:31.780894 5090 consensus_queue.cc:237] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:31.784992 1666 catalog_manager.cc:5484] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:31.806797 5134 raft_consensus.cc:482] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:31.806978 5134 raft_consensus.cc:504] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:31.807951 5134 leader_election.cc:288] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:31.819248 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7fd8dc7434b8ea14b9deb0023b4cb" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:31.819577 4569 raft_consensus.cc:2454] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:31.820263 4900 leader_election.cc:302] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:31.820564 5134 raft_consensus.cc:2781] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:31.820698 5134 raft_consensus.cc:482] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:31.820788 5134 raft_consensus.cc:3037] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.822846 5134 raft_consensus.cc:504] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:31.823522 5134 leader_election.cc:288] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:31.824107 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7fd8dc7434b8ea14b9deb0023b4cb" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:31.824226 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7fd8dc7434b8ea14b9deb0023b4cb" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:31.824360 1943 raft_consensus.cc:2454] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:31.824532 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7fd8dc7434b8ea14b9deb0023b4cb" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:31.824784 4569 raft_consensus.cc:3037] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:31.827723 4569 raft_consensus.cc:2454] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:31.828240 4900 leader_election.cc:302] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:12:31.828572 5134 raft_consensus.cc:2781] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:31.828756 5134 raft_consensus.cc:686] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:31.829035 5134 consensus_queue.cc:237] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:31.831928 1666 catalog_manager.cc:5484] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:31.928797 1670 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:31.940001 1813 raft_consensus.cc:1261] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:31.940560 4569 raft_consensus.cc:1261] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:31.941694 5090 consensus_queue.cc:1035] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:31.944433 5257 consensus_queue.cc:1035] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:31.955559 1668 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet e2da49afae5843b5a45e9ebaa67f3d0c (table rebalancer_dynamic_table [id=f08a2224751b42ecaee59ed46747f235]). Expected version 1 got 0
I20240503 08:12:31.968567 5301 tablet.cc:1722] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:31.969298 5301 tablet.cc:1620] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:31.974016 1665 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet e2da49afae5843b5a45e9ebaa67f3d0c (table rebalancer_dynamic_table [id=f08a2224751b42ecaee59ed46747f235]). Expected version 1 got 0
I20240503 08:12:31.978364 1666 catalog_manager.cc:5576] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181) does not have the latest schema for tablet e2da49afae5843b5a45e9ebaa67f3d0c (table rebalancer_dynamic_table [id=f08a2224751b42ecaee59ed46747f235]). Expected version 1 got 0
I20240503 08:12:31.988286 5303 tablet.cc:1722] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:31.989012 5303 tablet.cc:1620] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:31.992705 1666 catalog_manager.cc:5576] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181) does not have the latest schema for tablet e2da49afae5843b5a45e9ebaa67f3d0c (table rebalancer_dynamic_table [id=f08a2224751b42ecaee59ed46747f235]). Expected version 1 got 0
I20240503 08:12:31.997736 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=f08a2224751b42ecaee59ed46747f235] alter complete (version 1)
I20240503 08:12:32.002902 5297 tablet.cc:1722] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:32.013693 4885 consensus_queue.cc:1035] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.023751 5297 tablet.cc:1620] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:32.033602 5253 consensus_queue.cc:1035] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.038517 5190 consensus_queue.cc:1035] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.086426 5312 consensus_queue.cc:1035] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.106799 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:32.130429 4947 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:32.146878 2184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:12:32.147869 1923 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:12:32.185808 4549 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:32.206530 2054 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 08:12:32.208899 5297 tablet.cc:1722] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:32.209478 5297 tablet.cc:1620] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:32.209724 1665 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 08:12:32.210014 1665 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 08:12:32.211076 5301 tablet.cc:1722] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:32.211609 5301 tablet.cc:1620] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:32.218317 1665 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 7729383eb4b046eca0a4e7ff4c636ea0
I20240503 08:12:32.219249 1665 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 310d28bf7a30441b84b1e6bcf06e7619
I20240503 08:12:32.220717 1665 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 595f7bc77944474db016cdb39965765b
I20240503 08:12:32.222681 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 595f7bc77944474db016cdb39965765b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:32.223338 5060 tablet_replica.cc:330] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:32.223570 5060 raft_consensus.cc:2229] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.223903 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet 595f7bc77944474db016cdb39965765b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:32.224236 5213 tablet_replica.cc:330] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:32.224381 5060 raft_consensus.cc:2258] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.224478 5213 raft_consensus.cc:2229] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:32.224826 5213 raft_consensus.cc:2258] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.224870 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 310d28bf7a30441b84b1e6bcf06e7619 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:32.225169 5060 ts_tablet_manager.cc:1875] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.225596 5213 ts_tablet_manager.cc:1875] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.222676 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 7729383eb4b046eca0a4e7ff4c636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:32.227190 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=f08a2224751b42ecaee59ed46747f235] alter complete (version 2)
I20240503 08:12:32.227937 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet 7729383eb4b046eca0a4e7ff4c636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:32.228075 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 310d28bf7a30441b84b1e6bcf06e7619 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:32.228261 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet 595f7bc77944474db016cdb39965765b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:32.228566 5067 tablet_replica.cc:330] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:32.228801 5067 raft_consensus.cc:2229] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.229029 5067 raft_consensus.cc:2258] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.229729 5067 ts_tablet_manager.cc:1875] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.231609 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 310d28bf7a30441b84b1e6bcf06e7619 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:32.231875 5068 tablet_replica.cc:330] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:32.232120 5068 raft_consensus.cc:2229] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.232352 5068 raft_consensus.cc:2258] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.232718 5213 ts_tablet_manager.cc:1888] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.232874 5213 log.cc:1196] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/595f7bc77944474db016cdb39965765b
I20240503 08:12:32.233335 5213 ts_tablet_manager.cc:1909] T 595f7bc77944474db016cdb39965765b P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:32.234396 5213 tablet_replica.cc:330] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:32.234663 5213 raft_consensus.cc:2229] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:32.235034 5213 raft_consensus.cc:2258] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.235880 5213 ts_tablet_manager.cc:1875] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.238699 5067 ts_tablet_manager.cc:1888] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.239138 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 595f7bc77944474db016cdb39965765b (table rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]) successfully deleted
I20240503 08:12:32.241382 4682 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:32.251746 5060 ts_tablet_manager.cc:1888] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.252050 5060 log.cc:1196] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/595f7bc77944474db016cdb39965765b
I20240503 08:12:32.252298 5213 ts_tablet_manager.cc:1888] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.252458 5213 log.cc:1196] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/310d28bf7a30441b84b1e6bcf06e7619
I20240503 08:12:32.252905 5213 ts_tablet_manager.cc:1909] T 310d28bf7a30441b84b1e6bcf06e7619 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:32.253311 5060 ts_tablet_manager.cc:1909] T 595f7bc77944474db016cdb39965765b P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:32.254629 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 595f7bc77944474db016cdb39965765b (table rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]) successfully deleted
I20240503 08:12:32.255065 5060 tablet_replica.cc:330] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:32.255410 5067 log.cc:1196] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/7729383eb4b046eca0a4e7ff4c636ea0
I20240503 08:12:32.244376 5303 tablet.cc:1722] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:32.255770 1793 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:12:32.255897 5067 ts_tablet_manager.cc:1909] T 7729383eb4b046eca0a4e7ff4c636ea0 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:32.256175 5303 tablet.cc:1620] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:32.256997 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 7729383eb4b046eca0a4e7ff4c636ea0 (table rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]) successfully deleted
I20240503 08:12:32.257139 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 310d28bf7a30441b84b1e6bcf06e7619 (table rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]) successfully deleted
I20240503 08:12:32.259359 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 7729383eb4b046eca0a4e7ff4c636ea0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:32.261847 5067 tablet_replica.cc:330] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:32.263276 5067 raft_consensus.cc:2229] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.263617 5067 raft_consensus.cc:2258] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.263029 5060 raft_consensus.cc:2229] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:32.264737 5060 raft_consensus.cc:2258] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.265610 5060 ts_tablet_manager.cc:1875] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.274468 5067 ts_tablet_manager.cc:1875] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.282932 5060 ts_tablet_manager.cc:1888] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.300635 4814 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 08:12:32.301017 5060 log.cc:1196] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/7729383eb4b046eca0a4e7ff4c636ea0
I20240503 08:12:32.301728 5060 ts_tablet_manager.cc:1909] T 7729383eb4b046eca0a4e7ff4c636ea0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:32.302870 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 7729383eb4b046eca0a4e7ff4c636ea0 (table rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]) successfully deleted
I20240503 08:12:32.304623 5067 ts_tablet_manager.cc:1888] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.304774 5067 log.cc:1196] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/310d28bf7a30441b84b1e6bcf06e7619
I20240503 08:12:32.305301 5067 ts_tablet_manager.cc:1909] T 310d28bf7a30441b84b1e6bcf06e7619 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:32.308004 5068 ts_tablet_manager.cc:1875] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.313105 1943 raft_consensus.cc:3037] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.314554 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 310d28bf7a30441b84b1e6bcf06e7619 (table rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]) successfully deleted
I20240503 08:12:32.314986 5067 tablet_replica.cc:330] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:32.315223 5067 raft_consensus.cc:2229] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.315526 5067 raft_consensus.cc:2258] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.317842 5134 consensus_queue.cc:1035] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.320760 5067 ts_tablet_manager.cc:1875] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.322307 5068 ts_tablet_manager.cc:1888] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.322463 5068 log.cc:1196] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/310d28bf7a30441b84b1e6bcf06e7619
I20240503 08:12:32.322885 5068 ts_tablet_manager.cc:1909] T 310d28bf7a30441b84b1e6bcf06e7619 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:32.323868 5068 tablet_replica.cc:330] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:32.324081 5068 raft_consensus.cc:2229] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.324317 5068 raft_consensus.cc:2258] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.325043 5068 ts_tablet_manager.cc:1875] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.325803 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 310d28bf7a30441b84b1e6bcf06e7619 (table rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]) successfully deleted
I20240503 08:12:32.333405 1666 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:32.339278 5067 ts_tablet_manager.cc:1888] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.339463 5067 log.cc:1196] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/595f7bc77944474db016cdb39965765b
I20240503 08:12:32.339917 5067 ts_tablet_manager.cc:1909] T 595f7bc77944474db016cdb39965765b P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:32.341908 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 595f7bc77944474db016cdb39965765b (table rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]) successfully deleted
I20240503 08:12:32.348474 5297 tablet.cc:1722] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:32.349748 5301 tablet.cc:1722] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:32.349802 5303 tablet.cc:1722] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:32.359465 1666 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=f08a2224751b42ecaee59ed46747f235] alter complete (version 3)
I20240503 08:12:32.379174 5068 ts_tablet_manager.cc:1888] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.389151 5134 consensus_queue.cc:1035] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.392177 5068 log.cc:1196] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/7729383eb4b046eca0a4e7ff4c636ea0
I20240503 08:12:32.392843 5068 ts_tablet_manager.cc:1909] T 7729383eb4b046eca0a4e7ff4c636ea0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:32.398228 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 7729383eb4b046eca0a4e7ff4c636ea0 (table rebalance_test_table_8 [id=f26b7d1f07364f2789326fba2a9c062a]) successfully deleted
I20240503 08:12:32.431636 1666 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:32.442433 4549 tablet_service.cc:1467] Processing CreateTablet for tablet f0c5c7ecb94e453f85d81b36974902ef (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.442914 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c5c7ecb94e453f85d81b36974902ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.443068 4814 tablet_service.cc:1467] Processing CreateTablet for tablet f0c5c7ecb94e453f85d81b36974902ef (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.443560 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c5c7ecb94e453f85d81b36974902ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.447785 1793 tablet_service.cc:1467] Processing CreateTablet for tablet f0c5c7ecb94e453f85d81b36974902ef (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.448225 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c5c7ecb94e453f85d81b36974902ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.448845 4548 tablet_service.cc:1467] Processing CreateTablet for tablet f161c8da3b0e4b12b85ca0420a86dc07 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.449390 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f161c8da3b0e4b12b85ca0420a86dc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.452270 4813 tablet_service.cc:1467] Processing CreateTablet for tablet f161c8da3b0e4b12b85ca0420a86dc07 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.452785 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f161c8da3b0e4b12b85ca0420a86dc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.457526 1923 tablet_service.cc:1467] Processing CreateTablet for tablet f161c8da3b0e4b12b85ca0420a86dc07 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.457988 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f161c8da3b0e4b12b85ca0420a86dc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.461102 4812 tablet_service.cc:1467] Processing CreateTablet for tablet dab60fefb4704ef19490fb3c76bf47ab (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.461696 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab60fefb4704ef19490fb3c76bf47ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.464397 4947 tablet_service.cc:1467] Processing CreateTablet for tablet dab60fefb4704ef19490fb3c76bf47ab (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.464560 1666 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:32.464830 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab60fefb4704ef19490fb3c76bf47ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.464805 1666 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:32.475834 4548 tablet_service.cc:1467] Processing CreateTablet for tablet dab60fefb4704ef19490fb3c76bf47ab (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:32.476449 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab60fefb4704ef19490fb3c76bf47ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.476733 5340 tablet_bootstrap.cc:492] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:32.480005 5340 tablet_bootstrap.cc:654] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.483769 1666 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet e2da49afae5843b5a45e9ebaa67f3d0c
I20240503 08:12:32.485993 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet e2da49afae5843b5a45e9ebaa67f3d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:32.486418 4547 tablet_service.cc:1514] Processing DeleteTablet for tablet e2da49afae5843b5a45e9ebaa67f3d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:32.486822 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet e2da49afae5843b5a45e9ebaa67f3d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:32.487229 5067 tablet_replica.cc:330] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:32.487571 5067 raft_consensus.cc:2229] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.487919 5067 raft_consensus.cc:2258] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.489379 5067 ts_tablet_manager.cc:1875] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.521783 5067 ts_tablet_manager.cc:1888] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:32.523180 5067 log.cc:1196] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/e2da49afae5843b5a45e9ebaa67f3d0c
I20240503 08:12:32.523746 5067 ts_tablet_manager.cc:1909] T e2da49afae5843b5a45e9ebaa67f3d0c P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:32.524947 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet e2da49afae5843b5a45e9ebaa67f3d0c (table rebalancer_dynamic_table_new_name [id=f08a2224751b42ecaee59ed46747f235]) successfully deleted
I20240503 08:12:32.531198 5346 tablet_replica.cc:330] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:32.531507 5346 raft_consensus.cc:2229] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.531795 5346 raft_consensus.cc:2258] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.532125 5344 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:32.532644 5345 tablet_replica.cc:330] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:32.532965 5345 raft_consensus.cc:2229] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:32.533149 5346 ts_tablet_manager.cc:1875] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.534991 5344 tablet_bootstrap.cc:654] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.535658 5345 raft_consensus.cc:2258] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.535843 5347 tablet_bootstrap.cc:492] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:32.536899 5345 ts_tablet_manager.cc:1875] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.538141 5347 tablet_bootstrap.cc:654] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.542116 5340 tablet_bootstrap.cc:492] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:32.542284 5340 ts_tablet_manager.cc:1367] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.003s
I20240503 08:12:32.543179 5340 raft_consensus.cc:348] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:32.543437 5340 raft_consensus.cc:374] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.543524 5340 raft_consensus.cc:729] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.543836 5340 consensus_queue.cc:260] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:32.544836 5341 tablet_bootstrap.cc:492] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:32.545590 5342 tablet_bootstrap.cc:492] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:32.545836 5340 ts_tablet_manager.cc:1398] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:32.546713 5347 tablet_bootstrap.cc:492] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:32.546929 5347 ts_tablet_manager.cc:1367] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:12:32.547093 5340 tablet_bootstrap.cc:492] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:32.550179 5347 raft_consensus.cc:348] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:32.552335 5347 raft_consensus.cc:374] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.552520 5347 raft_consensus.cc:729] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.552834 5347 consensus_queue.cc:260] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:32.553362 5341 tablet_bootstrap.cc:654] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.553918 5347 ts_tablet_manager.cc:1398] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:32.555146 5342 tablet_bootstrap.cc:654] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.557755 5345 ts_tablet_manager.cc:1888] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:32.560022 5346 ts_tablet_manager.cc:1888] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:32.564889 5341 tablet_bootstrap.cc:492] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:32.564889 5344 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:32.565155 5344 ts_tablet_manager.cc:1367] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240503 08:12:32.565184 5341 ts_tablet_manager.cc:1367] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.005s
I20240503 08:12:32.566079 5341 raft_consensus.cc:348] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:32.566316 5344 raft_consensus.cc:348] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:32.566610 5344 raft_consensus.cc:374] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.566675 5341 raft_consensus.cc:374] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.566756 5344 raft_consensus.cc:729] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.566828 5341 raft_consensus.cc:729] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.567081 5344 consensus_queue.cc:260] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:32.567476 5341 consensus_queue.cc:260] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:32.569196 5341 ts_tablet_manager.cc:1398] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:32.570454 5342 tablet_bootstrap.cc:492] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:32.570667 5342 ts_tablet_manager.cc:1367] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.000s
I20240503 08:12:32.571636 5342 raft_consensus.cc:348] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:32.571959 5342 raft_consensus.cc:374] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.572145 5342 raft_consensus.cc:729] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.572487 5342 consensus_queue.cc:260] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:32.573755 5342 ts_tablet_manager.cc:1398] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:32.574108 5342 tablet_bootstrap.cc:492] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:32.579314 5342 tablet_bootstrap.cc:654] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.585062 5344 ts_tablet_manager.cc:1398] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240503 08:12:32.563747 5340 tablet_bootstrap.cc:654] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.585469 5346 log.cc:1196] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/e2da49afae5843b5a45e9ebaa67f3d0c
I20240503 08:12:32.589074 5346 ts_tablet_manager.cc:1909] T e2da49afae5843b5a45e9ebaa67f3d0c P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:32.589442 5342 tablet_bootstrap.cc:492] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:32.589641 5342 ts_tablet_manager.cc:1367] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20240503 08:12:32.590559 5342 raft_consensus.cc:348] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:32.590804 5342 raft_consensus.cc:374] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.590888 5342 raft_consensus.cc:729] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.590799 1666 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:32.591207 5342 consensus_queue.cc:260] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:32.592613 5342 ts_tablet_manager.cc:1398] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:32.592937 5342 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:32.593381 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet e2da49afae5843b5a45e9ebaa67f3d0c (table rebalancer_dynamic_table_new_name [id=f08a2224751b42ecaee59ed46747f235]) successfully deleted
I20240503 08:12:32.597244 5342 tablet_bootstrap.cc:654] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.601660 5345 log.cc:1196] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/e2da49afae5843b5a45e9ebaa67f3d0c
I20240503 08:12:32.602461 5345 ts_tablet_manager.cc:1909] T e2da49afae5843b5a45e9ebaa67f3d0c P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:32.604596 4947 tablet_service.cc:1467] Processing CreateTablet for tablet 29037649c190471b807f8459866ebcfa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e7714e7e0c7548a3b2c6f8a4015aabfc]), partition=
I20240503 08:12:32.604601 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 29037649c190471b807f8459866ebcfa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e7714e7e0c7548a3b2c6f8a4015aabfc]), partition=
I20240503 08:12:32.604995 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29037649c190471b807f8459866ebcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.605002 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29037649c190471b807f8459866ebcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.608047 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 29037649c190471b807f8459866ebcfa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e7714e7e0c7548a3b2c6f8a4015aabfc]), partition=
I20240503 08:12:32.608461 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29037649c190471b807f8459866ebcfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:32.619621 5344 tablet_bootstrap.cc:492] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:32.621915 5344 tablet_bootstrap.cc:654] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.626971 5340 tablet_bootstrap.cc:492] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:32.628448 5340 ts_tablet_manager.cc:1367] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.003s
I20240503 08:12:32.631563 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet e2da49afae5843b5a45e9ebaa67f3d0c (table rebalancer_dynamic_table_new_name [id=f08a2224751b42ecaee59ed46747f235]) successfully deleted
I20240503 08:12:32.631225 5340 raft_consensus.cc:348] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:32.632625 5340 raft_consensus.cc:374] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.632763 5340 raft_consensus.cc:729] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.633105 5340 consensus_queue.cc:260] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:32.636615 5340 ts_tablet_manager.cc:1398] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:12:32.639745 5340 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:32.647537 5190 raft_consensus.cc:482] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:32.647779 5190 raft_consensus.cc:504] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:32.648736 5190 leader_election.cc:288] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:32.654628 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f161c8da3b0e4b12b85ca0420a86dc07" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:32.654904 1943 raft_consensus.cc:2454] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:32.655536 4769 leader_election.cc:302] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:32.656147 5190 raft_consensus.cc:2781] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:32.656276 5190 raft_consensus.cc:482] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:32.656368 5190 raft_consensus.cc:3037] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.665545 5190 raft_consensus.cc:504] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:32.667490 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f161c8da3b0e4b12b85ca0420a86dc07" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:32.667798 1942 raft_consensus.cc:3037] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.667837 5190 leader_election.cc:288] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:32.674458 5342 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:32.674649 5342 ts_tablet_manager.cc:1367] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.006s
I20240503 08:12:32.674737 5344 tablet_bootstrap.cc:492] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:32.675384 1942 raft_consensus.cc:2454] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:32.676265 4769 leader_election.cc:302] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:32.676715 5312 raft_consensus.cc:2781] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:32.675879 5344 ts_tablet_manager.cc:1367] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.056s user 0.001s sys 0.008s
I20240503 08:12:32.686261 5344 raft_consensus.cc:348] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:32.686611 5344 raft_consensus.cc:374] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.686712 5344 raft_consensus.cc:729] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.687021 5344 consensus_queue.cc:260] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:32.676255 5342 raft_consensus.cc:348] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:32.690155 5342 raft_consensus.cc:374] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.690554 5342 raft_consensus.cc:729] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.691470 5342 consensus_queue.cc:260] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:32.692915 5342 ts_tablet_manager.cc:1398] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
W20240503 08:12:32.696067 1990 tablet.cc:2369] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:32.703604 5134 raft_consensus.cc:482] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:32.703862 5134 raft_consensus.cc:504] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:32.696483 5362 raft_consensus.cc:482] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:32.704738 5134 leader_election.cc:288] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:32.704926 5134 raft_consensus.cc:482] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:32.705073 5134 raft_consensus.cc:504] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:32.705737 5134 leader_election.cc:288] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:32.706147 5362 raft_consensus.cc:504] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:32.707481 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29037649c190471b807f8459866ebcfa" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:32.707073 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab60fefb4704ef19490fb3c76bf47ab" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:32.707984 4835 raft_consensus.cc:2454] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:32.709702 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab60fefb4704ef19490fb3c76bf47ab" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:32.709769 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29037649c190471b807f8459866ebcfa" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
W20240503 08:12:32.710543 4903 leader_election.cc:341] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:32.710894 4903 leader_election.cc:302] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:32.711205 5337 raft_consensus.cc:2781] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:32.711277 5337 raft_consensus.cc:482] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:32.712178 5337 raft_consensus.cc:3037] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.715317 5055 raft_consensus.cc:482] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:32.715513 5055 raft_consensus.cc:504] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
W20240503 08:12:32.715710 4900 leader_election.cc:341] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:12:32.716105 4900 leader_election.cc:341] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:32.716238 4900 leader_election.cc:302] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:32.716221 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f161c8da3b0e4b12b85ca0420a86dc07" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:32.716490 4568 raft_consensus.cc:2454] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:32.716845 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f161c8da3b0e4b12b85ca0420a86dc07" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:32.717113 1942 raft_consensus.cc:2379] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f95b05d111e4090bbc59dec394ed0ab in current term 1: Already voted for candidate f24648c92c384f1ba349eee801f985e2 in this term.
I20240503 08:12:32.717027 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f161c8da3b0e4b12b85ca0420a86dc07" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:32.716257 5055 leader_election.cc:288] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:32.717804 5134 raft_consensus.cc:2726] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:32.715276 5337 raft_consensus.cc:504] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:32.718740 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f161c8da3b0e4b12b85ca0420a86dc07" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:32.718968 4568 raft_consensus.cc:3037] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.719460 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab60fefb4704ef19490fb3c76bf47ab" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:32.719775 4834 raft_consensus.cc:3037] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.722023 4568 raft_consensus.cc:2454] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:32.722692 4834 raft_consensus.cc:2454] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:32.723383 4903 leader_election.cc:302] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:32.723572 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c5c7ecb94e453f85d81b36974902ef" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:32.723950 4568 raft_consensus.cc:2454] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:32.724187 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab60fefb4704ef19490fb3c76bf47ab" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:32.724828 4767 leader_election.cc:302] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:32.725180 5190 raft_consensus.cc:2781] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:32.725347 5190 raft_consensus.cc:482] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:32.725529 5190 raft_consensus.cc:3037] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:12:32.725911 4900 leader_election.cc:341] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:32.726222 5134 raft_consensus.cc:2781] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:32.727416 5337 leader_election.cc:288] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:32.728701 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c5c7ecb94e453f85d81b36974902ef" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:32.728971 1813 raft_consensus.cc:2454] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:32.732712 1672 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 08:12:32.733157 1672 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 08:12:32.734335 5344 ts_tablet_manager.cc:1398] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.049s user 0.001s sys 0.002s
I20240503 08:12:32.738126 1672 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 660769ce396e4c3ca80ac6c506755c55
I20240503 08:12:32.738775 1672 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 6a30093e839f49e6b25b639d24bea0a5
I20240503 08:12:32.739231 1672 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet af299410e1064415bb06bb045f93659d
I20240503 08:12:32.739630 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a30093e839f49e6b25b639d24bea0a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:32.740135 5060 tablet_replica.cc:330] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:32.740969 5060 raft_consensus.cc:2229] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.741308 5060 raft_consensus.cc:2258] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.741533 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet 660769ce396e4c3ca80ac6c506755c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:32.741621 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a30093e839f49e6b25b639d24bea0a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:32.741894 5067 tablet_replica.cc:330] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:32.742110 5067 raft_consensus.cc:2229] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:32.742183 5060 ts_tablet_manager.cc:1875] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.742633 5067 raft_consensus.cc:2258] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.743444 5067 ts_tablet_manager.cc:1875] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.743737 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 660769ce396e4c3ca80ac6c506755c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:32.744015 5068 tablet_replica.cc:330] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:32.744267 5068 raft_consensus.cc:2229] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.744503 5068 raft_consensus.cc:2258] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.745280 5068 ts_tablet_manager.cc:1875] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.745198 5190 raft_consensus.cc:504] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:32.748185 5190 leader_election.cc:288] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:32.748255 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet af299410e1064415bb06bb045f93659d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:32.748476 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet af299410e1064415bb06bb045f93659d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:32.748620 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c5c7ecb94e453f85d81b36974902ef" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:32.748626 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c5c7ecb94e453f85d81b36974902ef" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:32.748883 4569 raft_consensus.cc:3037] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.748896 1813 raft_consensus.cc:3037] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.750882 5067 ts_tablet_manager.cc:1888] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.751119 5067 log.cc:1196] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/660769ce396e4c3ca80ac6c506755c55
I20240503 08:12:32.751730 5067 ts_tablet_manager.cc:1909] T 660769ce396e4c3ca80ac6c506755c55 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:32.752431 1813 raft_consensus.cc:2454] T f0c5c7ecb94e453f85d81b36974902ef P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:32.753098 4769 leader_election.cc:302] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f581390da449ccb9e299011296405e, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:32.753571 5190 raft_consensus.cc:2781] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:32.753717 5190 raft_consensus.cc:686] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:32.754668 5362 leader_election.cc:288] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:32.754563 5190 consensus_queue.cc:237] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:32.755422 5312 raft_consensus.cc:686] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:32.755781 5067 tablet_replica.cc:330] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:32.755774 5312 consensus_queue.cc:237] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:32.756632 5067 raft_consensus.cc:2229] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.757491 5067 raft_consensus.cc:2258] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.759002 5067 ts_tablet_manager.cc:1875] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.759558 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 660769ce396e4c3ca80ac6c506755c55 (table rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]) successfully deleted
I20240503 08:12:32.760627 4569 raft_consensus.cc:2454] T f0c5c7ecb94e453f85d81b36974902ef P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:32.766474 5068 ts_tablet_manager.cc:1888] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.766707 5068 log.cc:1196] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/660769ce396e4c3ca80ac6c506755c55
I20240503 08:12:32.767155 5068 ts_tablet_manager.cc:1909] T 660769ce396e4c3ca80ac6c506755c55 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:32.768497 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 660769ce396e4c3ca80ac6c506755c55 (table rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]) successfully deleted
I20240503 08:12:32.769304 5060 ts_tablet_manager.cc:1888] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.769467 5060 log.cc:1196] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/6a30093e839f49e6b25b639d24bea0a5
I20240503 08:12:32.770041 5060 ts_tablet_manager.cc:1909] T 6a30093e839f49e6b25b639d24bea0a5 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:32.787590 5067 ts_tablet_manager.cc:1888] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.788005 4504 leader_election.cc:302] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, f24648c92c384f1ba349eee801f985e2
I20240503 08:12:32.788511 5068 tablet_replica.cc:330] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:32.788578 5055 raft_consensus.cc:2726] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:32.788844 5068 raft_consensus.cc:2229] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:32.789328 5068 raft_consensus.cc:2258] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.789085 1672 catalog_manager.cc:5484] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:32.792788 5067 log.cc:1196] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/6a30093e839f49e6b25b639d24bea0a5
I20240503 08:12:32.794121 5134 raft_consensus.cc:686] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:32.794561 5134 consensus_queue.cc:237] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:32.796461 1672 catalog_manager.cc:5484] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:32.798424 2139 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.194:41975, user_credentials={real_user=slave}} blocked reactor thread for 140051us
I20240503 08:12:32.800117 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 6a30093e839f49e6b25b639d24bea0a5 (table rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]) successfully deleted
I20240503 08:12:32.801949 5340 tablet_bootstrap.cc:654] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.801986 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet af299410e1064415bb06bb045f93659d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:32.802850 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 660769ce396e4c3ca80ac6c506755c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:32.803464 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a30093e839f49e6b25b639d24bea0a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:32 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:32.805696 5067 ts_tablet_manager.cc:1909] T 6a30093e839f49e6b25b639d24bea0a5 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:32.807596 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 6a30093e839f49e6b25b639d24bea0a5 (table rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]) successfully deleted
I20240503 08:12:32.810714 5067 tablet_replica.cc:330] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:32.811684 5067 raft_consensus.cc:2229] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.811774 1672 catalog_manager.cc:5484] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:32.812719 5067 raft_consensus.cc:2258] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.813535 5067 ts_tablet_manager.cc:1875] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.816170 5068 ts_tablet_manager.cc:1875] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.823655 5340 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:32.823966 5340 ts_tablet_manager.cc:1367] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.184s user 0.008s sys 0.002s
I20240503 08:12:32.825094 5340 raft_consensus.cc:348] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:32.825449 5340 raft_consensus.cc:374] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.825592 5340 raft_consensus.cc:729] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.825947 5340 consensus_queue.cc:260] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:32.830036 5067 ts_tablet_manager.cc:1888] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.832576 5340 ts_tablet_manager.cc:1398] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 08:12:32.834425 5372 tablet_replica.cc:330] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:32.835722 5372 raft_consensus.cc:2229] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.836083 5372 raft_consensus.cc:2258] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.836894 5340 tablet_bootstrap.cc:492] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:32.839473 5340 tablet_bootstrap.cc:654] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.839481 5068 ts_tablet_manager.cc:1888] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.839607 5068 log.cc:1196] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/af299410e1064415bb06bb045f93659d
I20240503 08:12:32.840986 5355 tablet_bootstrap.cc:492] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:32.842329 5372 ts_tablet_manager.cc:1875] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.842785 5068 ts_tablet_manager.cc:1909] T af299410e1064415bb06bb045f93659d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:32.842986 5355 tablet_bootstrap.cc:654] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:32.843946 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet af299410e1064415bb06bb045f93659d (table rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]) successfully deleted
I20240503 08:12:32.847508 5067 log.cc:1196] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/af299410e1064415bb06bb045f93659d
I20240503 08:12:32.848299 5067 ts_tablet_manager.cc:1909] T af299410e1064415bb06bb045f93659d P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:32.848363 5340 tablet_bootstrap.cc:492] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:32.848574 5340 ts_tablet_manager.cc:1367] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240503 08:12:32.848657 5355 tablet_bootstrap.cc:492] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:32.848891 5355 ts_tablet_manager.cc:1367] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20240503 08:12:32.849432 5340 raft_consensus.cc:348] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:32.849678 5340 raft_consensus.cc:374] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.849769 5340 raft_consensus.cc:729] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.849998 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet af299410e1064415bb06bb045f93659d (table rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]) successfully deleted
I20240503 08:12:32.849952 5355 raft_consensus.cc:348] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:32.850152 5355 raft_consensus.cc:374] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:32.850245 5355 raft_consensus.cc:729] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:32.850148 5340 consensus_queue.cc:260] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:32.850570 5355 consensus_queue.cc:260] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:32.854374 5355 ts_tablet_manager.cc:1398] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:32.855388 5340 ts_tablet_manager.cc:1398] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 08:12:32.862421 5055 raft_consensus.cc:482] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:32.862677 5055 raft_consensus.cc:504] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:32.863822 5055 leader_election.cc:288] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:32.866490 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab60fefb4704ef19490fb3c76bf47ab" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:32.866989 4835 raft_consensus.cc:2379] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f95b05d111e4090bbc59dec394ed0ab in current term 1: Already voted for candidate 00b61625d8a7460892c118a2570afda5 in this term.
I20240503 08:12:32.872260 5372 ts_tablet_manager.cc:1888] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.872455 5372 log.cc:1196] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/af299410e1064415bb06bb045f93659d
I20240503 08:12:32.876046 5372 ts_tablet_manager.cc:1909] T af299410e1064415bb06bb045f93659d P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:32.878566 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet af299410e1064415bb06bb045f93659d (table rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]) successfully deleted
I20240503 08:12:32.879014 5372 tablet_replica.cc:330] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:32.879313 5372 raft_consensus.cc:2229] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:32.879638 5372 raft_consensus.cc:2258] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.880441 5372 ts_tablet_manager.cc:1875] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.888514 2074 consensus_queue.cc:237] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:32.895201 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dab60fefb4704ef19490fb3c76bf47ab" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:32.896381 4502 leader_election.cc:302] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab; no voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2
I20240503 08:12:32.897051 5055 raft_consensus.cc:3037] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.903029 5372 ts_tablet_manager.cc:1888] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.903375 5372 log.cc:1196] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/660769ce396e4c3ca80ac6c506755c55
I20240503 08:12:32.903968 5372 ts_tablet_manager.cc:1909] T 660769ce396e4c3ca80ac6c506755c55 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:32.905211 5372 tablet_replica.cc:330] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:32.905341 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 660769ce396e4c3ca80ac6c506755c55 (table rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]) successfully deleted
I20240503 08:12:32.905493 5372 raft_consensus.cc:2229] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:32.905910 5372 raft_consensus.cc:2258] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:32.906827 5372 ts_tablet_manager.cc:1875] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:32.912191 5055 raft_consensus.cc:2726] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:32.914613 5372 ts_tablet_manager.cc:1888] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:32.914834 5372 log.cc:1196] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/6a30093e839f49e6b25b639d24bea0a5
I20240503 08:12:32.914882 1813 raft_consensus.cc:1261] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:32.915385 5372 ts_tablet_manager.cc:1909] T 6a30093e839f49e6b25b639d24bea0a5 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:32.915405 1943 raft_consensus.cc:1261] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:32.915971 4885 consensus_queue.cc:1035] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:32.916360 5359 consensus_queue.cc:1035] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:32.919646 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 6a30093e839f49e6b25b639d24bea0a5 (table rebalance_test_table_10 [id=f1acf111b6bf49d39e0f38ca4a22f3bf]) successfully deleted
I20240503 08:12:32.921711 5359 raft_consensus.cc:2932] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:32.923751 1943 raft_consensus.cc:2932] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:32.924010 1813 raft_consensus.cc:2932] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:32.930322 1672 catalog_manager.cc:5484] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:32.932345 2010 consensus_peers.cc:485] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3b2ebb6eee649199297eb13faa9e4cb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:32.950021 2074 consensus_queue.cc:237] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:32.952939 2203 raft_consensus.cc:1261] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:32.953019 1943 raft_consensus.cc:1261] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:32.953991 5359 consensus_queue.cc:1035] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:32.954376 4885 consensus_queue.cc:1035] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:32.957813 4885 raft_consensus.cc:2932] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:32.958498 1943 raft_consensus.cc:2932] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
W20240503 08:12:32.960718 2007 consensus_peers.cc:485] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48f411ec0bfc4a318b661441c49bd6ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:32.960407 2203 raft_consensus.cc:2932] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:32.966406 1666 catalog_manager.cc:5484] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:32.967296 5055 raft_consensus.cc:482] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:32.967494 5055 raft_consensus.cc:504] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:32.968290 5055 leader_election.cc:288] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:32.969390 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29037649c190471b807f8459866ebcfa" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:32.969640 4702 raft_consensus.cc:2454] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:32.969821 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29037649c190471b807f8459866ebcfa" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:32.970047 4967 raft_consensus.cc:2454] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:32.970078 4504 leader_election.cc:302] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:32.970371 5055 raft_consensus.cc:2781] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:32.970494 5055 raft_consensus.cc:482] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:32.970613 5055 raft_consensus.cc:3037] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.973100 5055 raft_consensus.cc:504] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:32.973760 5055 leader_election.cc:288] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:32.974292 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29037649c190471b807f8459866ebcfa" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:32.974325 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29037649c190471b807f8459866ebcfa" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:32.974515 4967 raft_consensus.cc:3037] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.974514 4702 raft_consensus.cc:3037] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:32.977370 4967 raft_consensus.cc:2454] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:32.978029 4502 leader_election.cc:302] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:32.978091 4702 raft_consensus.cc:2454] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:32.979460 5055 raft_consensus.cc:2781] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:32.979967 5055 raft_consensus.cc:686] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:32.980372 5055 consensus_queue.cc:237] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:32.985486 1665 catalog_manager.cc:5484] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:32.996613 1813 consensus_queue.cc:237] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:33.000061 1943 raft_consensus.cc:1261] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:33.000778 2203 raft_consensus.cc:1261] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:33.001461 5330 consensus_queue.cc:1035] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:33.001892 5331 consensus_queue.cc:1035] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:33.012277 5330 raft_consensus.cc:2932] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
W20240503 08:12:33.013666 1747 consensus_peers.cc:485] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 868bcace806f4d32977b21c876c2362b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:33.016764 1943 raft_consensus.cc:2932] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:33.027544 2202 raft_consensus.cc:2932] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:33.027957 1665 catalog_manager.cc:5484] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:33.054491 2074 consensus_queue.cc:237] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:33.057070 1813 raft_consensus.cc:1261] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:33.057125 2203 raft_consensus.cc:1261] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:33.057662 2010 consensus_peers.cc:485] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1811bd652d2c4afb9f52bcd64ceb2acb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:33.057843 5359 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:33.058781 5357 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 08:12:33.061069 5357 raft_consensus.cc:2932] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:33.063258 1811 raft_consensus.cc:2932] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:33.067478 2203 raft_consensus.cc:2932] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:33.069506 1672 catalog_manager.cc:5484] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:33.115521 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:33.121814 4702 raft_consensus.cc:1261] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:33.122005 4967 raft_consensus.cc:1261] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:33.122390 5055 consensus_queue.cc:1035] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:33.122699 5398 consensus_queue.cc:1035] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:33.128440 1665 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 29037649c190471b807f8459866ebcfa (table rebalancer_dynamic_table [id=e7714e7e0c7548a3b2c6f8a4015aabfc]). Expected version 1 got 0
I20240503 08:12:33.128780 5425 mvcc.cc:205] Tried to move back new op lower bound from 7023509311981223936 to 7023509311415939072. Current Snapshot: MvccSnapshot[applied={T|T < 7023509311981223936}]
I20240503 08:12:33.132889 5429 tablet.cc:1722] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:33.135084 5429 tablet.cc:1620] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:33.140810 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=e7714e7e0c7548a3b2c6f8a4015aabfc] alter complete (version 1)
I20240503 08:12:33.153282 5312 consensus_queue.cc:1035] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:33.174974 5431 tablet.cc:1722] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:33.175691 5431 tablet.cc:1620] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:33.176039 1672 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 29037649c190471b807f8459866ebcfa (table rebalancer_dynamic_table [id=e7714e7e0c7548a3b2c6f8a4015aabfc]). Expected version 1 got 0
I20240503 08:12:33.183527 5362 consensus_queue.cc:1035] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:33.194778 1672 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 29037649c190471b807f8459866ebcfa (table rebalancer_dynamic_table [id=e7714e7e0c7548a3b2c6f8a4015aabfc]). Expected version 1 got 0
I20240503 08:12:33.203790 5362 consensus_queue.cc:1035] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:33.204607 5427 mvcc.cc:205] Tried to move back new op lower bound from 7023509311981223936 to 7023509311415939072. Current Snapshot: MvccSnapshot[applied={T|T < 7023509311981223936}]
I20240503 08:12:33.215724 1672 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 29037649c190471b807f8459866ebcfa (table rebalancer_dynamic_table [id=e7714e7e0c7548a3b2c6f8a4015aabfc]). Expected version 1 got 0
I20240503 08:12:33.216645 5190 consensus_queue.cc:1035] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:33.218662 5438 tablet.cc:1722] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:33.220091 5438 tablet.cc:1620] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:33.242359 1672 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 08:12:33.242641 1672 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 08:12:33.244382 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:33.254380 5431 tablet.cc:1722] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:33.254866 5431 tablet.cc:1620] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:33.257688 5438 tablet.cc:1722] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:33.258168 5438 tablet.cc:1620] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:33.258392 1672 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet de60c691fd194765a969697118c9b90b
I20240503 08:12:33.258890 1672 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 4 replicas of tablet d3b2ebb6eee649199297eb13faa9e4cb
I20240503 08:12:33.259932 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet de60c691fd194765a969697118c9b90b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:33.260272 5060 tablet_replica.cc:330] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:33.260520 5060 raft_consensus.cc:2229] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.260680 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b2ebb6eee649199297eb13faa9e4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:33.260699 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet de60c691fd194765a969697118c9b90b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:33.260794 5060 raft_consensus.cc:2258] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.260948 5067 tablet_replica.cc:330] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:33.261159 5067 raft_consensus.cc:2229] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.261497 5067 raft_consensus.cc:2258] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.261632 5060 ts_tablet_manager.cc:1875] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.261791 5068 tablet_replica.cc:330] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:33.262010 5068 raft_consensus.cc:2229] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:33.262349 5068 raft_consensus.cc:2258] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.263087 5068 ts_tablet_manager.cc:1875] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.263474 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet de60c691fd194765a969697118c9b90b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:33.264859 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b2ebb6eee649199297eb13faa9e4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:33.265146 5372 tablet_replica.cc:330] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:33.265414 5372 raft_consensus.cc:2229] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:33.265878 5372 raft_consensus.cc:2258] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.266553 5337 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:33.266973 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b2ebb6eee649199297eb13faa9e4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:33.267374 5067 ts_tablet_manager.cc:1875] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.268857 5372 ts_tablet_manager.cc:1875] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.274226 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=e7714e7e0c7548a3b2c6f8a4015aabfc] alter complete (version 2)
I20240503 08:12:33.276013 1672 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 8da7019189fe4b36bbf08cba59a586a5
I20240503 08:12:33.276959 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet 8da7019189fe4b36bbf08cba59a586a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:33.276959 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 8da7019189fe4b36bbf08cba59a586a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:33.277279 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 8da7019189fe4b36bbf08cba59a586a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:33.277527 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b2ebb6eee649199297eb13faa9e4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:33.280790 5060 ts_tablet_manager.cc:1888] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:33.280915 5068 ts_tablet_manager.cc:1888] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:33.280965 5060 log.cc:1196] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/de60c691fd194765a969697118c9b90b
I20240503 08:12:33.281126 5068 log.cc:1196] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/de60c691fd194765a969697118c9b90b
I20240503 08:12:33.281435 5060 ts_tablet_manager.cc:1909] T de60c691fd194765a969697118c9b90b P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:33.281654 5068 ts_tablet_manager.cc:1909] T de60c691fd194765a969697118c9b90b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:33.282399 5060 tablet_replica.cc:330] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:33.282611 5060 raft_consensus.cc:2229] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:33.282668 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet de60c691fd194765a969697118c9b90b (table rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]) successfully deleted
I20240503 08:12:33.282964 5060 raft_consensus.cc:2258] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.282938 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet de60c691fd194765a969697118c9b90b (table rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]) successfully deleted
I20240503 08:12:33.284175 5060 ts_tablet_manager.cc:1875] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.284193 5067 ts_tablet_manager.cc:1888] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:33.284404 5067 log.cc:1196] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/d3b2ebb6eee649199297eb13faa9e4cb
I20240503 08:12:33.285349 5067 ts_tablet_manager.cc:1909] T d3b2ebb6eee649199297eb13faa9e4cb P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:33.287103 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet d3b2ebb6eee649199297eb13faa9e4cb (table rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]) successfully deleted
I20240503 08:12:33.287654 5067 tablet_replica.cc:330] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:33.288331 5067 raft_consensus.cc:2229] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.289589 5068 tablet_replica.cc:330] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:33.290264 5068 raft_consensus.cc:2229] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.289899 5067 raft_consensus.cc:2258] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:12:33.290374 1659 catalog_manager.cc:4797] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): delete failed for tablet d3b2ebb6eee649199297eb13faa9e4cb because the tablet was not found. No further retry: Not found: Tablet not found: d3b2ebb6eee649199297eb13faa9e4cb
I20240503 08:12:33.291282 5067 ts_tablet_manager.cc:1875] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.291833 5068 raft_consensus.cc:2258] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.292640 5372 ts_tablet_manager.cc:1888] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:33.292817 5372 log.cc:1196] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/d3b2ebb6eee649199297eb13faa9e4cb
I20240503 08:12:33.293388 5068 ts_tablet_manager.cc:1875] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.295688 5429 tablet.cc:1722] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:33.296247 5429 tablet.cc:1620] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:33.298058 5372 ts_tablet_manager.cc:1909] T d3b2ebb6eee649199297eb13faa9e4cb P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:33.300112 5060 ts_tablet_manager.cc:1888] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:33.300386 5060 log.cc:1196] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/8da7019189fe4b36bbf08cba59a586a5
I20240503 08:12:33.300962 5060 ts_tablet_manager.cc:1909] T 8da7019189fe4b36bbf08cba59a586a5 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:33.304311 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet d3b2ebb6eee649199297eb13faa9e4cb (table rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]) successfully deleted
I20240503 08:12:33.304636 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 8da7019189fe4b36bbf08cba59a586a5 (table rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]) successfully deleted
I20240503 08:12:33.304944 5372 tablet_replica.cc:330] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:33.305141 5372 raft_consensus.cc:2229] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.305382 5372 raft_consensus.cc:2258] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.305434 5068 ts_tablet_manager.cc:1888] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:33.305584 5068 log.cc:1196] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/d3b2ebb6eee649199297eb13faa9e4cb
I20240503 08:12:33.306016 5068 ts_tablet_manager.cc:1909] T d3b2ebb6eee649199297eb13faa9e4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:33.306059 5372 ts_tablet_manager.cc:1875] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.307077 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet d3b2ebb6eee649199297eb13faa9e4cb (table rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]) successfully deleted
I20240503 08:12:33.309300 5067 ts_tablet_manager.cc:1888] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:33.309468 5067 log.cc:1196] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/de60c691fd194765a969697118c9b90b
I20240503 08:12:33.312655 5067 ts_tablet_manager.cc:1909] T de60c691fd194765a969697118c9b90b P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:33.317706 5134 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:33.319559 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet de60c691fd194765a969697118c9b90b (table rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]) successfully deleted
I20240503 08:12:33.320680 5067 tablet_replica.cc:330] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:33.320971 5067 raft_consensus.cc:2229] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.321256 5067 raft_consensus.cc:2258] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.322016 5067 ts_tablet_manager.cc:1875] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.322757 5372 ts_tablet_manager.cc:1888] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:33.322933 5372 log.cc:1196] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/8da7019189fe4b36bbf08cba59a586a5
I20240503 08:12:33.323607 5372 ts_tablet_manager.cc:1909] T 8da7019189fe4b36bbf08cba59a586a5 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:33.327241 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 8da7019189fe4b36bbf08cba59a586a5 (table rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]) successfully deleted
I20240503 08:12:33.331149 5067 ts_tablet_manager.cc:1888] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:33.331391 5067 log.cc:1196] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/8da7019189fe4b36bbf08cba59a586a5
I20240503 08:12:33.331907 5067 ts_tablet_manager.cc:1909] T 8da7019189fe4b36bbf08cba59a586a5 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:33.333294 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 8da7019189fe4b36bbf08cba59a586a5 (table rebalance_test_table_12 [id=b226fb18fc1f404ca9860f6b761a47b8]) successfully deleted
I20240503 08:12:33.376484 1666 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:33.386350 4814 tablet_service.cc:1467] Processing CreateTablet for tablet f6cff57835b846e6b48c08bef52f5b68 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6b5d1749a8944d598384a9bb153c5fc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.386874 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cff57835b846e6b48c08bef52f5b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.387873 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 6024c62a4b534502b397a54cc8baa573 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6b5d1749a8944d598384a9bb153c5fc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.388311 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6024c62a4b534502b397a54cc8baa573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.388338 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 6024c62a4b534502b397a54cc8baa573 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6b5d1749a8944d598384a9bb153c5fc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.388798 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6024c62a4b534502b397a54cc8baa573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.389076 4947 tablet_service.cc:1467] Processing CreateTablet for tablet d6658d7bcd634abd808637e474018965 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6b5d1749a8944d598384a9bb153c5fc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.389474 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6658d7bcd634abd808637e474018965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.390280 1791 tablet_service.cc:1467] Processing CreateTablet for tablet f6cff57835b846e6b48c08bef52f5b68 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6b5d1749a8944d598384a9bb153c5fc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.390720 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cff57835b846e6b48c08bef52f5b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.398692 2052 tablet_service.cc:1467] Processing CreateTablet for tablet d6658d7bcd634abd808637e474018965 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6b5d1749a8944d598384a9bb153c5fc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.399349 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6658d7bcd634abd808637e474018965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.400696 5469 tablet_bootstrap.cc:492] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:33.401513 5471 tablet_bootstrap.cc:492] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:33.403626 5471 tablet_bootstrap.cc:654] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.404659 4812 tablet_service.cc:1467] Processing CreateTablet for tablet d6658d7bcd634abd808637e474018965 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6b5d1749a8944d598384a9bb153c5fc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.404742 4682 tablet_service.cc:1467] Processing CreateTablet for tablet f6cff57835b846e6b48c08bef52f5b68 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6b5d1749a8944d598384a9bb153c5fc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.405112 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6658d7bcd634abd808637e474018965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.405171 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cff57835b846e6b48c08bef52f5b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.405256 1666 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:33.409318 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 6024c62a4b534502b397a54cc8baa573 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=6b5d1749a8944d598384a9bb153c5fc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.409816 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6024c62a4b534502b397a54cc8baa573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.413445 5471 tablet_bootstrap.cc:492] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:33.413714 5471 ts_tablet_manager.cc:1367] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240503 08:12:33.414615 5471 raft_consensus.cc:348] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:33.415021 5471 raft_consensus.cc:374] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.415176 5471 raft_consensus.cc:729] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.415580 5471 consensus_queue.cc:260] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:33.416489 5471 ts_tablet_manager.cc:1398] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:33.435130 5429 tablet.cc:1722] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:33.436172 5438 tablet.cc:1722] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:33.448994 5431 tablet.cc:1722] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:33.450873 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=e7714e7e0c7548a3b2c6f8a4015aabfc] alter complete (version 3)
I20240503 08:12:33.463807 5475 ts_tablet_manager.cc:902] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:33.471935 5472 tablet_bootstrap.cc:492] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:33.478305 5474 tablet_bootstrap.cc:492] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:33.478837 5472 tablet_bootstrap.cc:654] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.480527 5476 tablet_bootstrap.cc:492] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:33.482869 5477 ts_tablet_manager.cc:902] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:33.485145 5472 tablet_bootstrap.cc:492] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:33.485396 5472 ts_tablet_manager.cc:1367] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240503 08:12:33.486371 5472 raft_consensus.cc:348] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:33.487185 5470 tablet_bootstrap.cc:492] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:33.487758 5472 raft_consensus.cc:374] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.488059 5472 raft_consensus.cc:729] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.488564 5472 consensus_queue.cc:260] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:33.489733 5472 ts_tablet_manager.cc:1398] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:33.490880 5472 tablet_bootstrap.cc:492] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:33.492146 5469 tablet_bootstrap.cc:654] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.495563 5472 tablet_bootstrap.cc:654] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.497617 5474 tablet_bootstrap.cc:654] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.498755 5470 tablet_bootstrap.cc:654] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.499012 5472 tablet_bootstrap.cc:492] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:33.499790 5472 ts_tablet_manager.cc:1367] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240503 08:12:33.499967 5476 tablet_bootstrap.cc:654] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.501912 5475 tablet_copy_client.cc:308] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:41181
I20240503 08:12:33.506424 5469 tablet_bootstrap.cc:492] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:33.506731 5469 ts_tablet_manager.cc:1367] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.106s user 0.004s sys 0.000s
I20240503 08:12:33.507547 5469 raft_consensus.cc:348] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:33.507874 5469 raft_consensus.cc:374] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.508020 5469 raft_consensus.cc:729] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.508352 5469 consensus_queue.cc:260] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:33.509222 5469 ts_tablet_manager.cc:1398] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:33.509605 5469 tablet_bootstrap.cc:492] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:33.512801 5469 tablet_bootstrap.cc:654] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.524435 5470 tablet_bootstrap.cc:492] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:33.524762 5470 ts_tablet_manager.cc:1367] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.121s user 0.005s sys 0.000s
I20240503 08:12:33.525918 5470 raft_consensus.cc:348] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:33.526317 5470 raft_consensus.cc:374] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.526469 5470 raft_consensus.cc:729] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.526877 5470 consensus_queue.cc:260] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:33.527267 5476 tablet_bootstrap.cc:492] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:33.527698 5470 ts_tablet_manager.cc:1398] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:33.529934 5474 tablet_bootstrap.cc:492] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:33.530853 5472 raft_consensus.cc:348] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:33.532387 5472 raft_consensus.cc:374] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.532600 5472 raft_consensus.cc:729] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.534488 5472 consensus_queue.cc:260] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:33.538492 5472 ts_tablet_manager.cc:1398] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.038s user 0.000s sys 0.002s
I20240503 08:12:33.540412 5476 ts_tablet_manager.cc:1367] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.000s
I20240503 08:12:33.547405 1833 tablet_copy_service.cc:140] P b8f581390da449ccb9e299011296405e: Received BeginTabletCopySession request for tablet 868bcace806f4d32977b21c876c2362b from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:57509)
I20240503 08:12:33.547566 5474 ts_tablet_manager.cc:1367] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.069s user 0.000s sys 0.007s
I20240503 08:12:33.547709 1833 tablet_copy_service.cc:161] P b8f581390da449ccb9e299011296405e: Beginning new tablet copy session on tablet 868bcace806f4d32977b21c876c2362b from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:57509: session id = 00b61625d8a7460892c118a2570afda5-868bcace806f4d32977b21c876c2362b
I20240503 08:12:33.548642 5474 raft_consensus.cc:348] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:33.548924 5474 raft_consensus.cc:374] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.549012 5474 raft_consensus.cc:729] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.549268 5474 consensus_queue.cc:260] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:33.542593 5476 raft_consensus.cc:348] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:33.549999 5476 raft_consensus.cc:374] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.550137 5476 raft_consensus.cc:729] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.550433 5476 consensus_queue.cc:260] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:33.551816 5497 ts_tablet_manager.cc:902] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:33.550451 5477 tablet_copy_client.cc:308] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:33.552724 5476 ts_tablet_manager.cc:1398] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 08:12:33.553067 5476 tablet_bootstrap.cc:492] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:33.555491 5476 tablet_bootstrap.cc:654] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.557935 5474 ts_tablet_manager.cc:1398] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 08:12:33.558185 1663 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:33.558426 1663 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:33.563701 1663 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 29037649c190471b807f8459866ebcfa
I20240503 08:12:33.564925 4947 tablet_service.cc:1514] Processing DeleteTablet for tablet 29037649c190471b807f8459866ebcfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:33.565232 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet 29037649c190471b807f8459866ebcfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:33.566161 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet 29037649c190471b807f8459866ebcfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:33.577906 5476 tablet_bootstrap.cc:492] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:33.578341 5476 ts_tablet_manager.cc:1367] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.010s
I20240503 08:12:33.578450 1833 tablet_copy_source_session.cc:205] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:33.581391 5475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868bcace806f4d32977b21c876c2362b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.584497 5476 raft_consensus.cc:348] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:33.584810 5476 raft_consensus.cc:374] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.584887 5476 raft_consensus.cc:729] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.585170 5476 consensus_queue.cc:260] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:33.586105 5476 ts_tablet_manager.cc:1398] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:33.590628 5475 tablet_copy_client.cc:783] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:33.591116 5475 tablet_copy_client.cc:647] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:33.598285 4946 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 08:12:33.602643 5469 tablet_bootstrap.cc:492] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:33.604904 5475 tablet_copy_client.cc:523] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:33.605906 5507 tablet_replica.cc:330] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:33.606559 5506 tablet_replica.cc:330] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:33.607178 5508 tablet_replica.cc:330] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
W20240503 08:12:33.607975 2250 tablet.cc:2369] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:33.614249 5469 ts_tablet_manager.cc:1367] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.105s user 0.004s sys 0.000s
I20240503 08:12:33.618757 5507 raft_consensus.cc:2229] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:33.619364 5507 raft_consensus.cc:2258] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.621979 2184 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:33.623558 5508 raft_consensus.cc:2229] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.623879 5508 raft_consensus.cc:2258] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.614358 5475 tablet_bootstrap.cc:492] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:33.624621 5506 raft_consensus.cc:2229] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.624974 5506 raft_consensus.cc:2258] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.627184 5507 ts_tablet_manager.cc:1875] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.627453 5469 raft_consensus.cc:348] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:33.627835 5469 raft_consensus.cc:374] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.627957 5469 raft_consensus.cc:729] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.628306 5469 consensus_queue.cc:260] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:33.629168 5469 ts_tablet_manager.cc:1398] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 08:12:33.631016 5360 raft_consensus.cc:482] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:33.631187 5360 raft_consensus.cc:504] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:33.632462 5360 leader_election.cc:288] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:33.656767 5508 ts_tablet_manager.cc:1875] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.668402 5506 ts_tablet_manager.cc:1875] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.668466 1663 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:33.670197 1923 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:12:33.678318 1791 tablet_service.cc:1467] Processing CreateTablet for tablet bd1fc514502a474593dcb4a1f13cfd07 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=77f714d301504bf3b4a6e527676b2b50]), partition=
I20240503 08:12:33.678761 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1fc514502a474593dcb4a1f13cfd07. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:12:33.680752 2120 tablet.cc:2369] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:33.677876 4947 tablet_service.cc:1467] Processing CreateTablet for tablet bd1fc514502a474593dcb4a1f13cfd07 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=77f714d301504bf3b4a6e527676b2b50]), partition=
I20240503 08:12:33.684154 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1fc514502a474593dcb4a1f13cfd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.687572 4812 tablet_service.cc:1467] Processing CreateTablet for tablet bd1fc514502a474593dcb4a1f13cfd07 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=77f714d301504bf3b4a6e527676b2b50]), partition=
I20240503 08:12:33.687989 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1fc514502a474593dcb4a1f13cfd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.689075 5497 tablet_copy_client.cc:308] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:33.697427 5507 ts_tablet_manager.cc:1888] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:33.701654 5507 log.cc:1196] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/29037649c190471b807f8459866ebcfa
I20240503 08:12:33.702126 5474 tablet_bootstrap.cc:492] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:33.706013 5474 tablet_bootstrap.cc:654] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.706102 5507 ts_tablet_manager.cc:1909] T 29037649c190471b807f8459866ebcfa P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:33.709587 5472 tablet_bootstrap.cc:492] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:33.711449 5472 tablet_bootstrap.cc:654] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.711943 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 29037649c190471b807f8459866ebcfa (table rebalancer_dynamic_table_new_name [id=e7714e7e0c7548a3b2c6f8a4015aabfc]) successfully deleted
I20240503 08:12:33.715680 5471 tablet_bootstrap.cc:492] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:33.717358 5471 tablet_bootstrap.cc:654] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.723310 1793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:12:33.764730 5471 tablet_bootstrap.cc:492] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:33.765041 5471 ts_tablet_manager.cc:1367] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.002s
I20240503 08:12:33.765868 5471 raft_consensus.cc:348] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:33.766253 5471 raft_consensus.cc:374] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.766376 5471 raft_consensus.cc:729] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.766839 5471 consensus_queue.cc:260] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:33.769465 5471 ts_tablet_manager.cc:1398] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:33.778250 5312 raft_consensus.cc:482] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:33.778453 5312 raft_consensus.cc:504] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:33.779156 5312 leader_election.cc:288] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:33.779525 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6024c62a4b534502b397a54cc8baa573" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:33.779836 2072 raft_consensus.cc:2454] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:33.780591 2138 leader_election.cc:302] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:33.781519 1665 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 08:12:33.781811 1665 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 08:12:33.784078 5474 tablet_bootstrap.cc:492] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:33.784133 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cff57835b846e6b48c08bef52f5b68" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:33.784327 5474 ts_tablet_manager.cc:1367] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.004s
I20240503 08:12:33.784585 1813 raft_consensus.cc:2454] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:33.785284 4769 leader_election.cc:302] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f581390da449ccb9e299011296405e, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:33.785235 5474 raft_consensus.cc:348] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:33.785521 5474 raft_consensus.cc:374] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.785626 5474 raft_consensus.cc:729] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.785858 5312 raft_consensus.cc:2781] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:33.785964 5312 raft_consensus.cc:482] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.786052 5312 raft_consensus.cc:3037] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.785923 5474 consensus_queue.cc:260] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:33.788225 1665 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet f1cc728faa604b7cb9e250961e33f2df
I20240503 08:12:33.789283 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cc728faa604b7cb9e250961e33f2df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:33.789602 5372 tablet_replica.cc:330] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:33.789842 5372 raft_consensus.cc:2229] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:33.790213 5372 raft_consensus.cc:2258] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.791051 5372 ts_tablet_manager.cc:1875] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.790239 5312 raft_consensus.cc:504] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:33.792076 5362 raft_consensus.cc:482] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:33.792268 5362 raft_consensus.cc:504] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:33.792979 5362 leader_election.cc:288] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:33.794056 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6658d7bcd634abd808637e474018965" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:33.794296 4967 raft_consensus.cc:2454] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:33.795904 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cc728faa604b7cb9e250961e33f2df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:33.796160 1665 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet f5df898b258f40d882ecba23b34e1c34
I20240503 08:12:33.796487 5068 tablet_replica.cc:330] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:33.796736 5068 raft_consensus.cc:2229] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.797003 5068 raft_consensus.cc:2258] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.797793 5068 ts_tablet_manager.cc:1875] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.799510 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet f5df898b258f40d882ecba23b34e1c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:33.799845 1665 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet aa19fc0ea2104f17aa69583702203c3c
I20240503 08:12:33.800463 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet f5df898b258f40d882ecba23b34e1c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:33.800525 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet aa19fc0ea2104f17aa69583702203c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:33.800772 5067 tablet_replica.cc:330] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:33.800997 5067 raft_consensus.cc:2229] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.801235 5067 raft_consensus.cc:2258] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.801630 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet aa19fc0ea2104f17aa69583702203c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:33.802047 5067 ts_tablet_manager.cc:1875] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.803946 5474 ts_tablet_manager.cc:1398] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 08:12:33.805182 5472 tablet_bootstrap.cc:492] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:33.805194 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet aa19fc0ea2104f17aa69583702203c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:33.805335 5472 ts_tablet_manager.cc:1367] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.096s user 0.001s sys 0.011s
I20240503 08:12:33.805738 5372 ts_tablet_manager.cc:1888] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:33.805886 5372 log.cc:1196] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/f1cc728faa604b7cb9e250961e33f2df
I20240503 08:12:33.806356 5372 ts_tablet_manager.cc:1909] T f1cc728faa604b7cb9e250961e33f2df P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:33.806354 5472 raft_consensus.cc:348] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:33.806581 5472 raft_consensus.cc:374] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.806672 5472 raft_consensus.cc:729] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.806928 5472 consensus_queue.cc:260] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:33.807482 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet f1cc728faa604b7cb9e250961e33f2df (table rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]) successfully deleted
I20240503 08:12:33.807862 5372 tablet_replica.cc:330] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:33.808334 5472 ts_tablet_manager.cc:1398] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:33.808082 5372 raft_consensus.cc:2229] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.808646 5372 raft_consensus.cc:2258] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.809351 5372 ts_tablet_manager.cc:1875] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.812022 5067 ts_tablet_manager.cc:1888] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:33.812551 5067 log.cc:1196] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/f5df898b258f40d882ecba23b34e1c34
I20240503 08:12:33.813778 5067 ts_tablet_manager.cc:1909] T f5df898b258f40d882ecba23b34e1c34 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:33.814000 2094 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 1811bd652d2c4afb9f52bcd64ceb2acb from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:35383)
I20240503 08:12:33.814268 2094 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 1811bd652d2c4afb9f52bcd64ceb2acb from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:35383: session id = 31f41542452b4b1eb994ff4223d039fc-1811bd652d2c4afb9f52bcd64ceb2acb
I20240503 08:12:33.815536 2052 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:12:33.816192 5067 tablet_replica.cc:330] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:33.816437 5067 raft_consensus.cc:2229] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.816687 5067 raft_consensus.cc:2258] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.817112 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cff57835b846e6b48c08bef52f5b68" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:33.817445 1813 raft_consensus.cc:3037] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.817939 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet f5df898b258f40d882ecba23b34e1c34 (table rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]) successfully deleted
I20240503 08:12:33.817875 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cc728faa604b7cb9e250961e33f2df with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:33.818755 5312 leader_election.cc:288] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:33.822728 1813 raft_consensus.cc:2454] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:33.823340 4769 leader_election.cc:302] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f581390da449ccb9e299011296405e, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:33.823742 5312 raft_consensus.cc:2781] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:33.824268 5312 raft_consensus.cc:686] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:33.824978 5312 consensus_queue.cc:237] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:33.830963 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 48f411ec0bfc4a318b661441c49bd6ec from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:56593)
I20240503 08:12:33.831251 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 48f411ec0bfc4a318b661441c49bd6ec from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:56593: session id = 5f95b05d111e4090bbc59dec394ed0ab-48f411ec0bfc4a318b661441c49bd6ec
I20240503 08:12:33.832582 2094 tablet_copy_source_session.cc:205] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:33.834594 2093 tablet_copy_source_session.cc:205] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:33.835968 4548 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:33.836869 5517 raft_consensus.cc:2781] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:33.837024 5517 raft_consensus.cc:482] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.837157 5517 raft_consensus.cc:3037] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.838472 5497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f411ec0bfc4a318b661441c49bd6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.840587 5068 ts_tablet_manager.cc:1888] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:33.840708 5477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1811bd652d2c4afb9f52bcd64ceb2acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.840749 5068 log.cc:1196] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/f1cc728faa604b7cb9e250961e33f2df
I20240503 08:12:33.841200 5068 ts_tablet_manager.cc:1909] T f1cc728faa604b7cb9e250961e33f2df P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:33.843938 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cff57835b846e6b48c08bef52f5b68" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:33.844244 4702 raft_consensus.cc:2454] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:33.845048 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cff57835b846e6b48c08bef52f5b68" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:33.845285 4702 raft_consensus.cc:3037] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.846776 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet f5df898b258f40d882ecba23b34e1c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:33 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:33.848722 4702 raft_consensus.cc:2454] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:33.846660 5497 tablet_copy_client.cc:783] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:33.850159 5508 ts_tablet_manager.cc:1888] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
W20240503 08:12:33.850739 2138 consensus_peers.cc:485] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:33.850795 5497 tablet_copy_client.cc:647] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:33.854262 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6024c62a4b534502b397a54cc8baa573" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:33.854787 4702 raft_consensus.cc:2454] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:33.855356 5508 log.cc:1196] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/29037649c190471b807f8459866ebcfa
I20240503 08:12:33.856055 5508 ts_tablet_manager.cc:1909] T 29037649c190471b807f8459866ebcfa P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:33.858584 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet f1cc728faa604b7cb9e250961e33f2df (table rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]) successfully deleted
I20240503 08:12:33.866147 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 29037649c190471b807f8459866ebcfa (table rebalancer_dynamic_table_new_name [id=e7714e7e0c7548a3b2c6f8a4015aabfc]) successfully deleted
W20240503 08:12:33.866681 2009 consensus_peers.cc:485] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:33.867805 5068 tablet_replica.cc:330] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:33.868664 5517 raft_consensus.cc:504] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:33.870961 4768 leader_election.cc:302] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:33.871508 5362 raft_consensus.cc:2781] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:33.871685 5362 raft_consensus.cc:482] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.871804 5362 raft_consensus.cc:3037] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.872500 5477 tablet_copy_client.cc:783] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:33.872911 5477 tablet_copy_client.cc:647] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:33.874657 5362 raft_consensus.cc:504] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:33.875749 5362 leader_election.cc:288] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:33.877616 5068 raft_consensus.cc:2229] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.877997 5068 raft_consensus.cc:2258] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.882791 5068 ts_tablet_manager.cc:1875] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.883533 5497 tablet_copy_client.cc:523] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:33.884560 5517 leader_election.cc:288] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:33.885603 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6024c62a4b534502b397a54cc8baa573" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:33.885860 4702 raft_consensus.cc:3037] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.886313 1665 catalog_manager.cc:5484] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:33.888615 4636 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletCopyService.BeginTabletCopySession -> {remote=127.27.157.195:43117, user_credentials={real_user=slave}} blocked reactor thread for 48861.3us
I20240503 08:12:33.889139 4702 raft_consensus.cc:2454] T 6024c62a4b534502b397a54cc8baa573 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:33.890451 4682 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:33.817428 5067 ts_tablet_manager.cc:1875] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.892323 2140 leader_election.cc:302] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:33.892767 5517 raft_consensus.cc:2781] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:33.893566 5517 raft_consensus.cc:686] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:33.896135 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6024c62a4b534502b397a54cc8baa573" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:33.896661 2072 raft_consensus.cc:3037] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.897117 5506 ts_tablet_manager.cc:1888] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:33.897153 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6658d7bcd634abd808637e474018965" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:33.897333 5506 log.cc:1196] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/29037649c190471b807f8459866ebcfa
I20240503 08:12:33.897431 4967 raft_consensus.cc:3037] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.898092 5506 ts_tablet_manager.cc:1909] T 29037649c190471b807f8459866ebcfa P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:33.899641 5372 ts_tablet_manager.cc:1888] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:33.899783 5372 log.cc:1196] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/f5df898b258f40d882ecba23b34e1c34
I20240503 08:12:33.900017 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 29037649c190471b807f8459866ebcfa (table rebalancer_dynamic_table_new_name [id=e7714e7e0c7548a3b2c6f8a4015aabfc]) successfully deleted
I20240503 08:12:33.900269 5372 ts_tablet_manager.cc:1909] T f5df898b258f40d882ecba23b34e1c34 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:33.900372 4967 raft_consensus.cc:2454] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:33.900533 2072 raft_consensus.cc:2454] T 6024c62a4b534502b397a54cc8baa573 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:33.901078 4768 leader_election.cc:302] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:33.901309 5372 tablet_replica.cc:330] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:33.901511 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet f5df898b258f40d882ecba23b34e1c34 (table rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]) successfully deleted
I20240503 08:12:33.901546 5372 raft_consensus.cc:2229] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:33.901574 5312 raft_consensus.cc:2781] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:33.901278 5477 tablet_copy_client.cc:523] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:33.901957 5372 raft_consensus.cc:2258] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.902683 5312 raft_consensus.cc:686] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:33.902904 5372 ts_tablet_manager.cc:1875] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.903090 5312 consensus_queue.cc:237] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:33.903730 5517 consensus_queue.cc:237] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:33.906258 5497 tablet_bootstrap.cc:492] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:33.909287 5477 tablet_bootstrap.cc:492] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:33.912871 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6658d7bcd634abd808637e474018965" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:33.913144 2074 raft_consensus.cc:3037] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.915902 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6658d7bcd634abd808637e474018965" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:33.917320 5535 tablet_replica.cc:330] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:33.917371 2074 raft_consensus.cc:2454] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:33.917709 5535 raft_consensus.cc:2229] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:33.918195 5535 raft_consensus.cc:2258] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.919072 5535 ts_tablet_manager.cc:1875] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.922570 5067 ts_tablet_manager.cc:1888] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20240503 08:12:33.923843 4903 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.199:39455, user_credentials={real_user=slave}} blocked reactor thread for 70343.2us
I20240503 08:12:33.924971 1665 catalog_manager.cc:5484] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:33.927670 5067 log.cc:1196] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/aa19fc0ea2104f17aa69583702203c3c
I20240503 08:12:33.934370 4812 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:33.933094 1665 catalog_manager.cc:5484] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:33.947929 5067 ts_tablet_manager.cc:1909] T aa19fc0ea2104f17aa69583702203c3c P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:33.954617 5475 tablet_bootstrap.cc:492] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:33.955210 5475 tablet_bootstrap.cc:492] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:12:33.955549 5475 ts_tablet_manager.cc:1367] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.341s user 0.059s sys 0.012s
I20240503 08:12:33.969769 5362 raft_consensus.cc:482] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:33.970005 5362 raft_consensus.cc:504] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:33.970849 5362 leader_election.cc:288] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:33.971913 5535 ts_tablet_manager.cc:1888] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:33.972167 5535 log.cc:1196] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/f5df898b258f40d882ecba23b34e1c34
I20240503 08:12:33.972604 5067 tablet_replica.cc:330] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:33.972745 5535 ts_tablet_manager.cc:1909] T f5df898b258f40d882ecba23b34e1c34 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:33.973524 5067 raft_consensus.cc:2229] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.973842 5067 raft_consensus.cc:2258] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.973923 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet f5df898b258f40d882ecba23b34e1c34 (table rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]) successfully deleted
I20240503 08:12:33.974714 5067 ts_tablet_manager.cc:1875] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:33.974828 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet aa19fc0ea2104f17aa69583702203c3c (table rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]) successfully deleted
I20240503 08:12:33.984779 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1fc514502a474593dcb4a1f13cfd07" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:33.984968 5475 raft_consensus.cc:348] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:33.985328 5475 raft_consensus.cc:729] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:12:33.985239 1813 raft_consensus.cc:2454] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:33.985606 5475 consensus_queue.cc:260] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:33.986016 4769 leader_election.cc:302] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f581390da449ccb9e299011296405e, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:33.986380 5362 raft_consensus.cc:2781] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:33.986479 5362 raft_consensus.cc:482] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.986585 5362 raft_consensus.cc:3037] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.988835 5475 ts_tablet_manager.cc:1398] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240503 08:12:33.989289 5362 raft_consensus.cc:504] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:33.990079 1833 tablet_copy_service.cc:342] P b8f581390da449ccb9e299011296405e: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-868bcace806f4d32977b21c876c2362b received from {username='slave'} at 127.27.157.200:57509
I20240503 08:12:33.990319 1833 tablet_copy_service.cc:433] P b8f581390da449ccb9e299011296405e: ending tablet copy session 00b61625d8a7460892c118a2570afda5-868bcace806f4d32977b21c876c2362b on tablet 868bcace806f4d32977b21c876c2362b with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:12:33.991824 5362 leader_election.cc:288] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:33.995282 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1fc514502a474593dcb4a1f13cfd07" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:33.995651 1813 raft_consensus.cc:3037] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.004302 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1fc514502a474593dcb4a1f13cfd07" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:34.004570 4966 raft_consensus.cc:3037] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.007469 5372 ts_tablet_manager.cc:1888] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.007767 5372 log.cc:1196] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/aa19fc0ea2104f17aa69583702203c3c
I20240503 08:12:34.008045 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1fc514502a474593dcb4a1f13cfd07" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:34.009240 4966 raft_consensus.cc:2454] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:34.010715 5372 ts_tablet_manager.cc:1909] T aa19fc0ea2104f17aa69583702203c3c P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:34.012071 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet aa19fc0ea2104f17aa69583702203c3c (table rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]) successfully deleted
I20240503 08:12:34.012770 4768 leader_election.cc:302] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:34.013120 5362 raft_consensus.cc:2781] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:34.013646 5362 raft_consensus.cc:686] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:34.013983 5362 consensus_queue.cc:237] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:34.026508 4966 raft_consensus.cc:1203] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:34.015705 1813 raft_consensus.cc:2454] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:34.040207 1665 catalog_manager.cc:5484] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:34.050522 5068 ts_tablet_manager.cc:1888] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.057474 5477 tablet_bootstrap.cc:492] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:34.058127 5477 tablet_bootstrap.cc:492] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:12:34.058485 5477 ts_tablet_manager.cc:1367] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.149s user 0.050s sys 0.009s
I20240503 08:12:34.059099 5067 ts_tablet_manager.cc:1888] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.059471 5477 raft_consensus.cc:348] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:34.059856 5477 raft_consensus.cc:729] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:12:34.060156 5477 consensus_queue.cc:260] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:34.060998 5477 ts_tablet_manager.cc:1398] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:34.061875 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-1811bd652d2c4afb9f52bcd64ceb2acb received from {username='slave'} at 127.27.157.198:35383
I20240503 08:12:34.062084 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-1811bd652d2c4afb9f52bcd64ceb2acb on tablet 1811bd652d2c4afb9f52bcd64ceb2acb with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:34.061429 5067 log.cc:1196] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/f1cc728faa604b7cb9e250961e33f2df
I20240503 08:12:34.068040 5067 ts_tablet_manager.cc:1909] T f1cc728faa604b7cb9e250961e33f2df P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:34.073880 5068 log.cc:1196] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa19fc0ea2104f17aa69583702203c3c
I20240503 08:12:34.074561 5068 ts_tablet_manager.cc:1909] T aa19fc0ea2104f17aa69583702203c3c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:34.076787 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet f1cc728faa604b7cb9e250961e33f2df (table rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]) successfully deleted
I20240503 08:12:34.079982 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet aa19fc0ea2104f17aa69583702203c3c (table rebalance_test_table_14 [id=9f62a9f6fd824e2e9bb6bbba182f6f56]) successfully deleted
I20240503 08:12:34.141278 5497 tablet_bootstrap.cc:492] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:34.142009 5497 tablet_bootstrap.cc:492] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:34.142386 5497 ts_tablet_manager.cc:1367] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.236s user 0.059s sys 0.003s
I20240503 08:12:34.143606 5497 raft_consensus.cc:348] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:34.143954 5497 raft_consensus.cc:729] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:34.146289 5497 consensus_queue.cc:260] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:34.166481 5497 ts_tablet_manager.cc:1398] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.024s user 0.004s sys 0.017s
I20240503 08:12:34.167907 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-48f411ec0bfc4a318b661441c49bd6ec received from {username='slave'} at 127.27.157.197:56593
I20240503 08:12:34.168071 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-48f411ec0bfc4a318b661441c49bd6ec on tablet 48f411ec0bfc4a318b661441c49bd6ec with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:34.188535 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:34.220048 1813 raft_consensus.cc:1261] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:34.220835 5362 consensus_queue.cc:1035] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.227106 4966 raft_consensus.cc:1261] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:34.228035 5362 consensus_queue.cc:1035] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.228482 1665 catalog_manager.cc:5576] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181) does not have the latest schema for tablet bd1fc514502a474593dcb4a1f13cfd07 (table rebalancer_dynamic_table [id=77f714d301504bf3b4a6e527676b2b50]). Expected version 1 got 0
I20240503 08:12:34.243407 1665 catalog_manager.cc:5576] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181) does not have the latest schema for tablet bd1fc514502a474593dcb4a1f13cfd07 (table rebalancer_dynamic_table [id=77f714d301504bf3b4a6e527676b2b50]). Expected version 1 got 0
I20240503 08:12:34.248344 5547 tablet.cc:1722] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:34.251374 5547 tablet.cc:1620] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:34.254583 5549 tablet.cc:1722] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:34.255672 5549 tablet.cc:1620] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:34.261704 1665 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet bd1fc514502a474593dcb4a1f13cfd07 (table rebalancer_dynamic_table [id=77f714d301504bf3b4a6e527676b2b50]). Expected version 1 got 0
I20240503 08:12:34.266152 5312 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.269133 5554 tablet.cc:1722] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:34.271742 5554 tablet.cc:1620] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:34.280931 1663 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet bd1fc514502a474593dcb4a1f13cfd07 (table rebalancer_dynamic_table [id=77f714d301504bf3b4a6e527676b2b50]). Expected version 1 got 0
I20240503 08:12:34.291141 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=77f714d301504bf3b4a6e527676b2b50] alter complete (version 1)
I20240503 08:12:34.302320 5312 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.304143 1672 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 08:12:34.304603 1672 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 08:12:34.322682 1672 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet dc82c5ccee9e47e4bde9ff51860fd8a0
W20240503 08:12:34.323699 1878 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.196:42183, user_credentials={real_user=slave}} blocked reactor thread for 74814.7us
W20240503 08:12:34.325838 1879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.193:41181, user_credentials={real_user=slave}} blocked reactor thread for 69808.1us
I20240503 08:12:34.332024 5362 consensus_queue.cc:1035] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.338357 1672 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet fabf21f75e4944bebdf8da5581c4af34
I20240503 08:12:34.338891 1672 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 8d45993944a946adacf5dd05ec67e3b7
I20240503 08:12:34.340492 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet fabf21f75e4944bebdf8da5581c4af34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:34.340699 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d45993944a946adacf5dd05ec67e3b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:34.341054 5372 tablet_replica.cc:330] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:34.341353 5372 raft_consensus.cc:2229] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.341720 5372 raft_consensus.cc:2258] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.342718 5372 ts_tablet_manager.cc:1875] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.344259 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet dc82c5ccee9e47e4bde9ff51860fd8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:34.344592 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d45993944a946adacf5dd05ec67e3b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:34.344862 5535 tablet_replica.cc:330] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:34.345152 5535 raft_consensus.cc:2229] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:34.345613 5535 raft_consensus.cc:2258] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.346251 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet dc82c5ccee9e47e4bde9ff51860fd8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:34.346573 5535 ts_tablet_manager.cc:1875] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.346942 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet fabf21f75e4944bebdf8da5581c4af34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:34.347669 5068 tablet_replica.cc:330] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:34.347904 5068 raft_consensus.cc:2229] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.348197 5068 raft_consensus.cc:2258] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.348704 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet fabf21f75e4944bebdf8da5581c4af34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:34.348954 5068 ts_tablet_manager.cc:1875] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.348984 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d45993944a946adacf5dd05ec67e3b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:34.349226 5067 tablet_replica.cc:330] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:34.349440 5067 raft_consensus.cc:2229] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:34.349540 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet dc82c5ccee9e47e4bde9ff51860fd8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:34.349836 5067 raft_consensus.cc:2258] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.350662 5067 ts_tablet_manager.cc:1875] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.356251 2074 consensus_queue.cc:237] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:34.356793 5068 ts_tablet_manager.cc:1888] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.356954 5068 log.cc:1196] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/dc82c5ccee9e47e4bde9ff51860fd8a0
I20240503 08:12:34.357429 5068 ts_tablet_manager.cc:1909] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:34.358407 5068 tablet_replica.cc:330] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:34.358624 5068 raft_consensus.cc:2229] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.358873 5068 raft_consensus.cc:2258] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.359627 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet dc82c5ccee9e47e4bde9ff51860fd8a0 (table rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]) successfully deleted
I20240503 08:12:34.360579 5068 ts_tablet_manager.cc:1875] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.365747 5372 ts_tablet_manager.cc:1888] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.365993 5372 log.cc:1196] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/fabf21f75e4944bebdf8da5581c4af34
I20240503 08:12:34.366631 5372 ts_tablet_manager.cc:1909] T fabf21f75e4944bebdf8da5581c4af34 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:34.369524 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet fabf21f75e4944bebdf8da5581c4af34 (table rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]) successfully deleted
I20240503 08:12:34.369961 5372 tablet_replica.cc:330] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:34.370280 5372 raft_consensus.cc:2229] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.370599 5372 raft_consensus.cc:2258] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.371825 5372 ts_tablet_manager.cc:1875] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.373306 5068 ts_tablet_manager.cc:1888] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.373468 5068 log.cc:1196] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/fabf21f75e4944bebdf8da5581c4af34
I20240503 08:12:34.373951 5068 ts_tablet_manager.cc:1909] T fabf21f75e4944bebdf8da5581c4af34 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:34.375087 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet fabf21f75e4944bebdf8da5581c4af34 (table rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]) successfully deleted
I20240503 08:12:34.379001 5535 ts_tablet_manager.cc:1888] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.379382 5067 ts_tablet_manager.cc:1888] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.379658 5535 log.cc:1196] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/dc82c5ccee9e47e4bde9ff51860fd8a0
I20240503 08:12:34.381150 5372 ts_tablet_manager.cc:1888] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.381443 5372 log.cc:1196] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/8d45993944a946adacf5dd05ec67e3b7
I20240503 08:12:34.382123 5372 ts_tablet_manager.cc:1909] T 8d45993944a946adacf5dd05ec67e3b7 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:34.382799 5535 ts_tablet_manager.cc:1909] T dc82c5ccee9e47e4bde9ff51860fd8a0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:34.382994 5067 log.cc:1196] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/fabf21f75e4944bebdf8da5581c4af34
I20240503 08:12:34.383661 5067 ts_tablet_manager.cc:1909] T fabf21f75e4944bebdf8da5581c4af34 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:34.383775 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 8d45993944a946adacf5dd05ec67e3b7 (table rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]) successfully deleted
I20240503 08:12:34.385857 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet fabf21f75e4944bebdf8da5581c4af34 (table rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]) successfully deleted
I20240503 08:12:34.386441 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet dc82c5ccee9e47e4bde9ff51860fd8a0 (table rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]) successfully deleted
I20240503 08:12:34.386919 5067 tablet_replica.cc:330] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:34.387243 5067 raft_consensus.cc:2229] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.387506 5535 tablet_replica.cc:330] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:34.387602 5067 raft_consensus.cc:2258] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.387847 5535 raft_consensus.cc:2229] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:34.388265 5535 raft_consensus.cc:2258] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.388621 5067 ts_tablet_manager.cc:1875] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.389130 5535 ts_tablet_manager.cc:1875] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.389047 5312 consensus_queue.cc:1035] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.397665 1663 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:34.399053 2203 raft_consensus.cc:1261] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:34.400019 5357 consensus_queue.cc:1035] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:34.399986 5067 ts_tablet_manager.cc:1888] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.401294 5067 log.cc:1196] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/8d45993944a946adacf5dd05ec67e3b7
I20240503 08:12:34.401881 5067 ts_tablet_manager.cc:1909] T 8d45993944a946adacf5dd05ec67e3b7 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:34.403357 5067 tablet_replica.cc:330] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:34.403440 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 8d45993944a946adacf5dd05ec67e3b7 (table rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]) successfully deleted
I20240503 08:12:34.403628 5067 raft_consensus.cc:2229] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.403986 5067 raft_consensus.cc:2258] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.404997 5067 ts_tablet_manager.cc:1875] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:12:34.405509 2010 consensus_peers.cc:485] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 358cc7bd631d432385d6205db86cbfd4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:34.412567 1663 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet bd1fc514502a474593dcb4a1f13cfd07 (table rebalancer_dynamic_table [id=77f714d301504bf3b4a6e527676b2b50]). Expected version 2 got 1
I20240503 08:12:34.415447 5554 tablet.cc:1722] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:34.415750 5067 ts_tablet_manager.cc:1888] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.415927 5067 log.cc:1196] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc82c5ccee9e47e4bde9ff51860fd8a0
I20240503 08:12:34.415974 5554 tablet.cc:1620] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:34.416131 5547 tablet.cc:1722] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:34.416507 5549 tablet.cc:1722] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:34.416602 5547 tablet.cc:1620] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:34.416944 5549 tablet.cc:1620] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:34.419448 5535 ts_tablet_manager.cc:1888] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.419696 5535 log.cc:1196] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/8d45993944a946adacf5dd05ec67e3b7
I20240503 08:12:34.420253 5535 ts_tablet_manager.cc:1909] T 8d45993944a946adacf5dd05ec67e3b7 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:34.424685 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 8d45993944a946adacf5dd05ec67e3b7 (table rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]) successfully deleted
I20240503 08:12:34.427531 5360 consensus_queue.cc:1035] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.427623 5067 ts_tablet_manager.cc:1909] T dc82c5ccee9e47e4bde9ff51860fd8a0 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:34.430836 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet dc82c5ccee9e47e4bde9ff51860fd8a0 (table rebalance_test_table_16 [id=1c93a14622e34eecb8d6bf508be45494]) successfully deleted
I20240503 08:12:34.431783 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=77f714d301504bf3b4a6e527676b2b50] alter complete (version 2)
I20240503 08:12:34.431906 1943 raft_consensus.cc:1261] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:34.432641 5359 consensus_queue.cc:1035] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:34.444502 2203 raft_consensus.cc:2932] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:34.442893 5357 raft_consensus.cc:2932] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:34.456907 5517 consensus_queue.cc:1035] T 6024c62a4b534502b397a54cc8baa573 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.461573 1943 raft_consensus.cc:2932] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:34.471477 1663 catalog_manager.cc:5484] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:34.488868 2074 consensus_queue.cc:237] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:34.491451 1943 raft_consensus.cc:1261] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:34.491621 2203 raft_consensus.cc:1261] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:34.492038 2007 consensus_peers.cc:485] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89d9b1b3c23241d49bfcb914a5bd1777. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:34.492074 5359 consensus_queue.cc:1035] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:34.492470 5573 consensus_queue.cc:1035] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:34.495661 5573 raft_consensus.cc:2932] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:34.496905 1943 raft_consensus.cc:2932] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:34.497007 2203 raft_consensus.cc:2932] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:34.502163 5531 raft_consensus.cc:1050] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:12:34.501947 1665 catalog_manager.cc:5484] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:34.503254 5531 consensus_queue.cc:237] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:34.507864 4966 raft_consensus.cc:1261] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:34.508702 5407 consensus_queue.cc:1035] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:34.513922 1944 raft_consensus.cc:1261] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:34.514492 2203 raft_consensus.cc:1261] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:34.514855 5531 consensus_queue.cc:1035] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:34.515192 5407 consensus_queue.cc:1035] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:34.518827 5407 raft_consensus.cc:2932] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:34.523586 1944 raft_consensus.cc:2932] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:34.526365 4966 raft_consensus.cc:2932] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:34.530542 1665 catalog_manager.cc:5484] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:34.534097 2203 raft_consensus.cc:2932] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:34.538898 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:34.540019 1663 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:12:34.544983 5531 consensus_queue.cc:1159] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "00b61625d8a7460892c118a2570afda5" responder_term: 1 status { last_received { term: 1 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 5 } }
I20240503 08:12:34.545950 1813 consensus_queue.cc:237] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:34.549121 2201 raft_consensus.cc:1261] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:34.551216 5331 consensus_queue.cc:1035] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:34.555509 5547 tablet.cc:1722] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:34.564076 5549 tablet.cc:1722] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:34.565061 5407 raft_consensus.cc:2932] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:34.573201 2202 raft_consensus.cc:2932] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:34.575250 5554 tablet.cc:1722] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:34.579478 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 868bcace806f4d32977b21c876c2362b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:34.581059 2053 tablet_service.cc:1467] Processing CreateTablet for tablet f4900ffdbc624376b32f054efdb100b5 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.581158 1666 catalog_manager.cc:5484] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:34.581708 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4900ffdbc624376b32f054efdb100b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.588181 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 868bcace806f4d32977b21c876c2362b with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:34.588591 5068 tablet_replica.cc:330] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:34.588852 5068 raft_consensus.cc:2229] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.589190 5068 raft_consensus.cc:2258] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.589609 1923 tablet_service.cc:1467] Processing CreateTablet for tablet f4900ffdbc624376b32f054efdb100b5 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.590150 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4900ffdbc624376b32f054efdb100b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.590399 4947 tablet_service.cc:1467] Processing CreateTablet for tablet f4900ffdbc624376b32f054efdb100b5 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.590864 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4900ffdbc624376b32f054efdb100b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.593283 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 54fe406b5f324d629243f7f939535620 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.593780 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54fe406b5f324d629243f7f939535620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.593849 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 938428b201e842448232a1d77061afab (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.594278 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938428b201e842448232a1d77061afab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.595762 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 938428b201e842448232a1d77061afab (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.596192 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938428b201e842448232a1d77061afab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.596346 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 54fe406b5f324d629243f7f939535620 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.596755 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54fe406b5f324d629243f7f939535620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.602249 4568 raft_consensus.cc:1203] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:34.604677 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 54fe406b5f324d629243f7f939535620 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.605196 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54fe406b5f324d629243f7f939535620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.610980 5068 ts_tablet_manager.cc:1875] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:34.617178 4703 raft_consensus.cc:1203] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:34.617770 5597 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:34.622632 5597 tablet_bootstrap.cc:654] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.632519 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=77f714d301504bf3b4a6e527676b2b50] alter complete (version 3)
I20240503 08:12:34.635006 5597 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:34.635560 4966 raft_consensus.cc:1261] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:34.635476 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 938428b201e842448232a1d77061afab (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:34.638726 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938428b201e842448232a1d77061afab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.642683 5597 ts_tablet_manager.cc:1367] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20240503 08:12:34.649068 5531 consensus_queue.cc:1035] T 868bcace806f4d32977b21c876c2362b P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:34.662797 5068 ts_tablet_manager.cc:1888] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:34.663265 5068 log.cc:1196] T 868bcace806f4d32977b21c876c2362b P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/868bcace806f4d32977b21c876c2362b
I20240503 08:12:34.664570 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 868bcace806f4d32977b21c876c2362b (table rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]) successfully deleted
I20240503 08:12:34.665027 4965 raft_consensus.cc:2932] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:34.665745 5600 tablet_bootstrap.cc:492] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:34.668372 5600 tablet_bootstrap.cc:654] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.673521 5598 tablet_bootstrap.cc:492] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:34.677032 5598 tablet_bootstrap.cc:654] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.681928 5597 raft_consensus.cc:348] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:34.682191 5597 raft_consensus.cc:374] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.682289 5597 raft_consensus.cc:729] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.682518 5597 consensus_queue.cc:260] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:34.683094 5597 ts_tablet_manager.cc:1398] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.040s user 0.001s sys 0.000s
I20240503 08:12:34.688463 5596 tablet_bootstrap.cc:492] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:34.689567 5601 tablet_bootstrap.cc:492] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:34.694260 5601 tablet_bootstrap.cc:654] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.693905 5596 tablet_bootstrap.cc:654] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.700932 5600 tablet_bootstrap.cc:492] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:34.701010 5601 tablet_bootstrap.cc:492] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:34.701390 5601 ts_tablet_manager.cc:1367] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240503 08:12:34.701663 5600 ts_tablet_manager.cc:1367] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.005s
I20240503 08:12:34.701998 5598 tablet_bootstrap.cc:492] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:34.702291 5601 raft_consensus.cc:348] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:34.702647 5601 raft_consensus.cc:374] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.702761 5598 ts_tablet_manager.cc:1367] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240503 08:12:34.702766 5601 raft_consensus.cc:729] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.703013 5601 consensus_queue.cc:260] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:34.703677 5601 ts_tablet_manager.cc:1398] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:34.704051 5601 tablet_bootstrap.cc:492] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:34.705351 5600 raft_consensus.cc:348] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:34.706434 5601 tablet_bootstrap.cc:654] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.710110 5600 raft_consensus.cc:374] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.710866 5598 raft_consensus.cc:348] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:34.717337 5600 raft_consensus.cc:729] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.717687 5598 raft_consensus.cc:374] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.718071 5600 consensus_queue.cc:260] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:34.718453 5598 raft_consensus.cc:729] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.719271 5598 consensus_queue.cc:260] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:34.720080 5600 ts_tablet_manager.cc:1398] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240503 08:12:34.721071 2202 consensus_queue.cc:237] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:34.722193 5598 ts_tablet_manager.cc:1398] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240503 08:12:34.729182 5596 tablet_bootstrap.cc:492] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:34.729369 5596 ts_tablet_manager.cc:1367] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20240503 08:12:34.731014 5596 raft_consensus.cc:348] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:34.731266 5596 raft_consensus.cc:374] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.731400 5596 raft_consensus.cc:729] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.731710 5596 consensus_queue.cc:260] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:34.735955 5596 ts_tablet_manager.cc:1398] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:34.736407 1668 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:34.736644 1668 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:34.740588 1668 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet bd1fc514502a474593dcb4a1f13cfd07
I20240503 08:12:34.741541 4946 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1fc514502a474593dcb4a1f13cfd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:34.741851 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1fc514502a474593dcb4a1f13cfd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:34.742358 5067 tablet_replica.cc:330] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:34.742619 5067 raft_consensus.cc:2229] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.742882 5067 raft_consensus.cc:2258] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.744119 4812 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1fc514502a474593dcb4a1f13cfd07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:34.747781 5067 ts_tablet_manager.cc:1875] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.747923 5601 tablet_bootstrap.cc:492] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:34.748075 5601 ts_tablet_manager.cc:1367] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.023s
I20240503 08:12:34.748782 5601 raft_consensus.cc:348] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:34.748991 5601 raft_consensus.cc:374] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.749104 5601 raft_consensus.cc:729] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.749382 5601 consensus_queue.cc:260] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:34.750041 5601 ts_tablet_manager.cc:1398] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 08:12:34.750499 5601 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:34.752286 5601 tablet_bootstrap.cc:654] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
W20240503 08:12:34.756170 2140 consensus_peers.cc:485] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64973388b1c8466798b93ee424df489d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:34.757323 1944 raft_consensus.cc:1261] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:34.758008 5360 consensus_queue.cc:1035] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:34.758621 5609 tablet_replica.cc:330] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:34.759300 5609 raft_consensus.cc:2229] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:34.760041 5609 raft_consensus.cc:2258] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.761209 5609 ts_tablet_manager.cc:1875] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.762214 5601 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:34.762483 5601 ts_tablet_manager.cc:1367] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240503 08:12:34.762814 5067 ts_tablet_manager.cc:1888] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:34.762970 5067 log.cc:1196] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/bd1fc514502a474593dcb4a1f13cfd07
I20240503 08:12:34.763486 5067 ts_tablet_manager.cc:1909] T bd1fc514502a474593dcb4a1f13cfd07 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:34.764012 5601 raft_consensus.cc:348] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:34.764393 5601 raft_consensus.cc:374] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.764557 5601 raft_consensus.cc:729] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.764550 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet bd1fc514502a474593dcb4a1f13cfd07 (table rebalancer_dynamic_table_new_name [id=77f714d301504bf3b4a6e527676b2b50]) successfully deleted
I20240503 08:12:34.764926 5601 consensus_queue.cc:260] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:34.766867 5601 ts_tablet_manager.cc:1398] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:34.767443 5596 tablet_bootstrap.cc:492] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:34.767618 2074 raft_consensus.cc:1261] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:34.768321 5360 consensus_queue.cc:1035] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:34.779675 5596 tablet_bootstrap.cc:654] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.780906 5608 tablet_replica.cc:330] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:34.781271 5608 raft_consensus.cc:2229] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.781595 5608 raft_consensus.cc:2258] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.782676 5608 ts_tablet_manager.cc:1875] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.791972 5596 tablet_bootstrap.cc:492] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:34.792232 5596 ts_tablet_manager.cc:1367] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.001s
I20240503 08:12:34.791765 5360 raft_consensus.cc:2932] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:34.793793 5596 raft_consensus.cc:348] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:34.794129 5596 raft_consensus.cc:374] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.794270 5596 raft_consensus.cc:729] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.794606 5596 consensus_queue.cc:260] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:34.795445 5596 ts_tablet_manager.cc:1398] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:12:34.795691 5609 ts_tablet_manager.cc:1888] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:34.796399 5596 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:34.796185 1943 raft_consensus.cc:2932] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:34.797878 5608 ts_tablet_manager.cc:1888] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:34.798453 5596 tablet_bootstrap.cc:654] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.799412 5608 log.cc:1196] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/bd1fc514502a474593dcb4a1f13cfd07
I20240503 08:12:34.800280 5608 ts_tablet_manager.cc:1909] T bd1fc514502a474593dcb4a1f13cfd07 P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:34.801959 1668 catalog_manager.cc:5484] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:34.804662 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet bd1fc514502a474593dcb4a1f13cfd07 (table rebalancer_dynamic_table_new_name [id=77f714d301504bf3b4a6e527676b2b50]) successfully deleted
I20240503 08:12:34.812721 5553 raft_consensus.cc:482] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:34.812916 5553 raft_consensus.cc:504] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:34.814430 5553 leader_election.cc:288] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:34.815448 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4900ffdbc624376b32f054efdb100b5" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:34.815805 2074 raft_consensus.cc:2454] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:34.822064 1878 leader_election.cc:302] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:34.822424 5553 raft_consensus.cc:2781] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:34.822618 5553 raft_consensus.cc:482] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:34.822813 5553 raft_consensus.cc:3037] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.826256 5553 raft_consensus.cc:504] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:34.827148 5553 leader_election.cc:288] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:34.828023 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4900ffdbc624376b32f054efdb100b5" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:34.828249 2074 raft_consensus.cc:3037] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.831121 2074 raft_consensus.cc:2454] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:34.831703 5533 raft_consensus.cc:482] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:34.831912 5533 raft_consensus.cc:504] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:34.832062 1878 leader_election.cc:302] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:34.832435 5553 raft_consensus.cc:2781] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:34.832919 5553 raft_consensus.cc:686] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:34.832930 5533 leader_election.cc:288] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:34.833216 5553 consensus_queue.cc:237] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:34.835530 5573 raft_consensus.cc:482] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:34.835757 5573 raft_consensus.cc:504] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:34.836722 5573 leader_election.cc:288] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:34.837240 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938428b201e842448232a1d77061afab" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:34.837469 1813 raft_consensus.cc:2454] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:34.838003 2009 leader_election.cc:302] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:34.838292 5573 raft_consensus.cc:2781] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:34.838394 5573 raft_consensus.cc:482] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:34.838469 5573 raft_consensus.cc:3037] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.841414 5573 raft_consensus.cc:504] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:34.842890 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938428b201e842448232a1d77061afab" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:34.843128 1813 raft_consensus.cc:3037] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.845602 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:34.845945 1813 raft_consensus.cc:2454] T 938428b201e842448232a1d77061afab P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:34.846544 2009 leader_election.cc:302] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:34.846913 5357 raft_consensus.cc:2781] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:34.847090 5357 raft_consensus.cc:686] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:34.847388 5357 consensus_queue.cc:237] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:34.848862 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938428b201e842448232a1d77061afab" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:34.849123 4965 raft_consensus.cc:2454] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:34.849633 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938428b201e842448232a1d77061afab" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:34.849825 4965 raft_consensus.cc:3037] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.850414 5573 leader_election.cc:288] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:34.853022 4965 raft_consensus.cc:2454] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:34.859253 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=52811fa0cad54e73a98995e835065069]), partition=
I20240503 08:12:34.859426 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=52811fa0cad54e73a98995e835065069]), partition=
I20240503 08:12:34.859776 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.859897 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.862373 1668 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 08:12:34.862666 1668 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 08:12:34.866865 1665 catalog_manager.cc:5484] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:34.868141 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=52811fa0cad54e73a98995e835065069]), partition=
I20240503 08:12:34.868556 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:34.870345 5609 log.cc:1196] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/bd1fc514502a474593dcb4a1f13cfd07
I20240503 08:12:34.871198 5609 ts_tablet_manager.cc:1909] T bd1fc514502a474593dcb4a1f13cfd07 P f24648c92c384f1ba349eee801f985e2: Deleting consensus metadata
I20240503 08:12:34.844728 5362 raft_consensus.cc:482] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:34.875231 5362 raft_consensus.cc:504] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:34.876080 5362 leader_election.cc:288] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:34.877418 5596 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:34.877408 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fe406b5f324d629243f7f939535620" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:34.877648 5596 ts_tablet_manager.cc:1367] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.081s user 0.000s sys 0.027s
I20240503 08:12:34.878216 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fe406b5f324d629243f7f939535620" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:34.878437 4965 raft_consensus.cc:2454] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:34.878456 5596 raft_consensus.cc:348] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:34.878646 5596 raft_consensus.cc:374] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.878794 5596 raft_consensus.cc:729] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
W20240503 08:12:34.878855 4768 leader_election.cc:341] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:34.879088 5596 consensus_queue.cc:260] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:34.880206 4768 leader_election.cc:302] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters: 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:12:34.880283 5596 ts_tablet_manager.cc:1398] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:34.881466 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4900ffdbc624376b32f054efdb100b5" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:34.881511 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet bd1fc514502a474593dcb4a1f13cfd07 (table rebalancer_dynamic_table_new_name [id=77f714d301504bf3b4a6e527676b2b50]) successfully deleted
I20240503 08:12:34.881776 2074 raft_consensus.cc:2379] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00b61625d8a7460892c118a2570afda5 in current term 1: Already voted for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in this term.
I20240503 08:12:34.882905 5362 raft_consensus.cc:2781] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:34.883014 5362 raft_consensus.cc:482] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:34.883113 5362 raft_consensus.cc:3037] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.883011 1663 catalog_manager.cc:5484] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:34.884424 5624 tablet_bootstrap.cc:492] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:34.884475 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4900ffdbc624376b32f054efdb100b5" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:34.885205 4902 leader_election.cc:302] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:12:34.885658 5620 raft_consensus.cc:3037] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.886039 5624 tablet_bootstrap.cc:654] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.886564 5362 raft_consensus.cc:504] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:34.886783 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4900ffdbc624376b32f054efdb100b5" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:34.886862 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4900ffdbc624376b32f054efdb100b5" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:34.890511 5620 raft_consensus.cc:2726] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:34.890836 4965 raft_consensus.cc:2454] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:34.891624 5625 tablet_bootstrap.cc:492] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:34.893287 5625 tablet_bootstrap.cc:654] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.894196 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fe406b5f324d629243f7f939535620" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:34.894515 4966 raft_consensus.cc:3037] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.898272 4966 raft_consensus.cc:2454] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:34.898746 4768 leader_election.cc:302] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:34.898972 5624 tablet_bootstrap.cc:492] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:34.899107 5190 raft_consensus.cc:2781] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:34.899152 5624 ts_tablet_manager.cc:1367] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240503 08:12:34.899247 5190 raft_consensus.cc:686] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:34.899575 5190 consensus_queue.cc:237] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:34.900280 5624 raft_consensus.cc:348] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:34.900555 5624 raft_consensus.cc:374] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.900655 5624 raft_consensus.cc:729] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.900956 5624 consensus_queue.cc:260] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:34.901975 5624 ts_tablet_manager.cc:1398] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:34.904637 2072 raft_consensus.cc:2932] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:34.908592 1668 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 668fe552878f42bbb5156c5e162b2630
I20240503 08:12:34.909758 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 668fe552878f42bbb5156c5e162b2630 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:34.909821 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 668fe552878f42bbb5156c5e162b2630 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:34.910215 5068 tablet_replica.cc:330] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:34.910573 5068 raft_consensus.cc:2229] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.910627 1668 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet dbbf914fa37043e8874583d1fdc914b6
I20240503 08:12:34.910707 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet 668fe552878f42bbb5156c5e162b2630 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:34.910925 5068 raft_consensus.cc:2258] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.911020 5067 tablet_replica.cc:330] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:34.911234 5067 raft_consensus.cc:2229] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:34.911628 5067 raft_consensus.cc:2258] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.911779 5068 ts_tablet_manager.cc:1875] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.912184 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbf914fa37043e8874583d1fdc914b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:34.912345 5067 ts_tablet_manager.cc:1875] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.912817 1668 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet c00305bb59f84487b2afcdc30910a663
I20240503 08:12:34.912820 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbf914fa37043e8874583d1fdc914b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:34.913306 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbf914fa37043e8874583d1fdc914b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:34.913684 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet c00305bb59f84487b2afcdc30910a663 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:34.915517 5535 tablet_replica.cc:330] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:34.915758 5535 raft_consensus.cc:2229] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.916033 5535 raft_consensus.cc:2258] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.916828 5535 ts_tablet_manager.cc:1875] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.921715 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet c00305bb59f84487b2afcdc30910a663 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:34.922459 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet c00305bb59f84487b2afcdc30910a663 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:34 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:34.923154 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fe406b5f324d629243f7f939535620" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:34.923496 2074 raft_consensus.cc:3037] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:34.924677 5068 ts_tablet_manager.cc:1888] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.924957 5068 log.cc:1196] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/668fe552878f42bbb5156c5e162b2630
I20240503 08:12:34.925489 5068 ts_tablet_manager.cc:1909] T 668fe552878f42bbb5156c5e162b2630 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:34.926859 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 668fe552878f42bbb5156c5e162b2630 (table rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]) successfully deleted
I20240503 08:12:34.927661 5068 tablet_replica.cc:330] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:34.928328 5068 raft_consensus.cc:2229] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.929050 5068 raft_consensus.cc:2258] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.930060 5068 ts_tablet_manager.cc:1875] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.930455 2074 raft_consensus.cc:2454] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:34.931092 5535 ts_tablet_manager.cc:1888] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.931265 5535 log.cc:1196] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/dbbf914fa37043e8874583d1fdc914b6
I20240503 08:12:34.931960 5362 leader_election.cc:288] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:34.936632 5535 ts_tablet_manager.cc:1909] T dbbf914fa37043e8874583d1fdc914b6 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:34.937331 5067 ts_tablet_manager.cc:1888] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.938773 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet dbbf914fa37043e8874583d1fdc914b6 (table rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]) successfully deleted
I20240503 08:12:34.939072 5068 ts_tablet_manager.cc:1888] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.939220 5068 log.cc:1196] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/c00305bb59f84487b2afcdc30910a663
I20240503 08:12:34.939551 5067 log.cc:1196] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/668fe552878f42bbb5156c5e162b2630
I20240503 08:12:34.939843 5068 ts_tablet_manager.cc:1909] T c00305bb59f84487b2afcdc30910a663 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:34.940451 5067 ts_tablet_manager.cc:1909] T 668fe552878f42bbb5156c5e162b2630 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:34.941453 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet c00305bb59f84487b2afcdc30910a663 (table rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]) successfully deleted
I20240503 08:12:34.944406 5600 tablet_bootstrap.cc:492] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:34.944551 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 668fe552878f42bbb5156c5e162b2630 (table rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]) successfully deleted
I20240503 08:12:34.945765 5067 tablet_replica.cc:330] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:34.946010 5067 raft_consensus.cc:2229] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:34.946156 5600 tablet_bootstrap.cc:654] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:34.946370 5067 raft_consensus.cc:2258] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.949553 5067 ts_tablet_manager.cc:1875] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.951825 5600 tablet_bootstrap.cc:492] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:34.952232 5600 ts_tablet_manager.cc:1367] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20240503 08:12:34.953498 5600 raft_consensus.cc:348] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:34.953960 5600 raft_consensus.cc:374] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.954181 5600 raft_consensus.cc:729] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.954617 5600 consensus_queue.cc:260] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:34.958230 5600 ts_tablet_manager.cc:1398] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:34.960767 5630 tablet_replica.cc:330] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:34.961025 5630 raft_consensus.cc:2229] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.961545 5630 raft_consensus.cc:2258] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.963879 5630 ts_tablet_manager.cc:1875] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:34.966173 1665 catalog_manager.cc:5484] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:34.967983 5067 ts_tablet_manager.cc:1888] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.985002 5625 tablet_bootstrap.cc:492] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:34.985414 5625 ts_tablet_manager.cc:1367] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.094s user 0.002s sys 0.020s
I20240503 08:12:34.986655 5625 raft_consensus.cc:348] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:34.986919 5634 ts_tablet_manager.cc:902] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:34.986915 5625 raft_consensus.cc:374] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:34.987017 5625 raft_consensus.cc:729] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:34.987272 5625 consensus_queue.cc:260] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:34.987413 5067 log.cc:1196] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/dbbf914fa37043e8874583d1fdc914b6
I20240503 08:12:34.988003 5067 ts_tablet_manager.cc:1909] T dbbf914fa37043e8874583d1fdc914b6 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:34.988147 5625 ts_tablet_manager.cc:1398] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:34.989048 5067 tablet_replica.cc:330] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:34.989293 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet dbbf914fa37043e8874583d1fdc914b6 (table rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]) successfully deleted
I20240503 08:12:34.993849 5630 ts_tablet_manager.cc:1888] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:34.994014 5630 log.cc:1196] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/668fe552878f42bbb5156c5e162b2630
I20240503 08:12:34.994524 5630 ts_tablet_manager.cc:1909] T 668fe552878f42bbb5156c5e162b2630 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:34.995705 5067 raft_consensus.cc:2229] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:34.996011 5067 raft_consensus.cc:2258] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:34.996836 5067 ts_tablet_manager.cc:1875] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:35.002322 5067 ts_tablet_manager.cc:1888] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:35.002483 5067 log.cc:1196] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/c00305bb59f84487b2afcdc30910a663
I20240503 08:12:35.003551 5634 tablet_copy_client.cc:308] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:35.003824 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 668fe552878f42bbb5156c5e162b2630 (table rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]) successfully deleted
I20240503 08:12:35.004150 5067 ts_tablet_manager.cc:1909] T c00305bb59f84487b2afcdc30910a663 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:35.004257 5630 tablet_replica.cc:330] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:35.004595 5630 raft_consensus.cc:2229] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:35.004849 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 358cc7bd631d432385d6205db86cbfd4 from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:52311)
I20240503 08:12:35.005023 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 358cc7bd631d432385d6205db86cbfd4 from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:52311: session id = f24648c92c384f1ba349eee801f985e2-358cc7bd631d432385d6205db86cbfd4
I20240503 08:12:35.005416 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet c00305bb59f84487b2afcdc30910a663 (table rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]) successfully deleted
I20240503 08:12:35.005461 5630 raft_consensus.cc:2258] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:35.006358 5630 ts_tablet_manager.cc:1875] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:35.008261 2093 tablet_copy_source_session.cc:205] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:35.008479 4834 consensus_queue.cc:237] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:35.013568 5630 ts_tablet_manager.cc:1888] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:35.013842 5630 log.cc:1196] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/dbbf914fa37043e8874583d1fdc914b6
I20240503 08:12:35.014413 5630 ts_tablet_manager.cc:1909] T dbbf914fa37043e8874583d1fdc914b6 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:35.015921 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet dbbf914fa37043e8874583d1fdc914b6 (table rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]) successfully deleted
I20240503 08:12:35.017175 5630 tablet_replica.cc:330] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:35.017590 5630 raft_consensus.cc:2229] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:35.018097 5630 raft_consensus.cc:2258] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:35.019354 5630 ts_tablet_manager.cc:1875] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:35.026819 5630 ts_tablet_manager.cc:1888] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 08:12:35.026911 5359 raft_consensus.cc:1050] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:12:35.027046 5630 log.cc:1196] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/c00305bb59f84487b2afcdc30910a663
I20240503 08:12:35.027588 5630 ts_tablet_manager.cc:1909] T c00305bb59f84487b2afcdc30910a663 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:35.028776 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet c00305bb59f84487b2afcdc30910a663 (table rebalance_test_table_18 [id=7f74723fa74a43f590ce741d046208a5]) successfully deleted
I20240503 08:12:35.028472 5359 consensus_queue.cc:237] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:35.030465 5634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358cc7bd631d432385d6205db86cbfd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.035809 5640 ts_tablet_manager.cc:902] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:35.037667 2202 raft_consensus.cc:1261] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:35.037807 4966 raft_consensus.cc:1261] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:12:35.038455 4769 consensus_peers.cc:485] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 820ecfef09a34327a08f7c3a893ccb05. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:35.041653 2202 raft_consensus.cc:1261] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:35.041819 4703 raft_consensus.cc:1261] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:35.042277 5357 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:35.042627 5573 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:35.043197 1813 raft_consensus.cc:1261] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:35.044658 5357 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:35.047674 5357 raft_consensus.cc:2932] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:35.048513 5362 consensus_queue.cc:1035] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20240503 08:12:35.048290 4702 raft_consensus.cc:2932] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:35.049697 5190 consensus_queue.cc:1035] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:35.054503 1665 catalog_manager.cc:5484] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:35.061725 5634 tablet_copy_client.cc:783] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:35.063832 5634 tablet_copy_client.cc:647] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:35.064694 1813 raft_consensus.cc:2932] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:35.073123 2203 raft_consensus.cc:2932] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:35.078828 5543 raft_consensus.cc:482] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:35.079021 5543 raft_consensus.cc:504] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:35.080101 5543 leader_election.cc:288] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:35.079056 5190 raft_consensus.cc:2932] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:35.080943 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f7c4ff8fae42c4bedaa95a16e6f07c" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:35.081239 1813 raft_consensus.cc:2454] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:35.081241 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f7c4ff8fae42c4bedaa95a16e6f07c" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:35.081705 4503 leader_election.cc:302] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:35.081974 5543 raft_consensus.cc:2781] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:35.082021 4703 raft_consensus.cc:2454] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:35.082103 5543 raft_consensus.cc:482] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:35.082195 5543 raft_consensus.cc:3037] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.086256 5543 raft_consensus.cc:504] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:35.086894 4965 raft_consensus.cc:2932] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:35.088629 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f7c4ff8fae42c4bedaa95a16e6f07c" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:35.089470 4703 raft_consensus.cc:3037] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.092067 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f7c4ff8fae42c4bedaa95a16e6f07c" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:35.092325 1811 raft_consensus.cc:3037] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.095142 1811 raft_consensus.cc:2454] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:35.094913 1668 catalog_manager.cc:5484] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:35.096508 5634 tablet_copy_client.cc:523] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:35.104887 4503 leader_election.cc:302] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:35.105849 2202 raft_consensus.cc:2932] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:35.107695 5543 leader_election.cc:288] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:35.108155 5543 raft_consensus.cc:2781] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:35.109508 5634 tablet_bootstrap.cc:492] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:35.110343 5543 raft_consensus.cc:686] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:35.110920 5543 consensus_queue.cc:237] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:35.112825 4703 raft_consensus.cc:2454] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:35.119710 5640 tablet_copy_client.cc:308] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:35.121191 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 89d9b1b3c23241d49bfcb914a5bd1777 from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:56593)
I20240503 08:12:35.121378 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 89d9b1b3c23241d49bfcb914a5bd1777 from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:56593: session id = 5f95b05d111e4090bbc59dec394ed0ab-89d9b1b3c23241d49bfcb914a5bd1777
I20240503 08:12:35.124405 2093 tablet_copy_source_session.cc:205] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:35.132848 5640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d9b1b3c23241d49bfcb914a5bd1777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.142340 1665 catalog_manager.cc:5484] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:35.152561 5573 raft_consensus.cc:1050] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:35.153383 5573 consensus_queue.cc:237] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:35.154925 5640 tablet_copy_client.cc:783] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:35.155242 5640 tablet_copy_client.cc:647] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:35.156400 4568 raft_consensus.cc:1261] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:35.156435 1943 raft_consensus.cc:1261] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:35.157166 5640 tablet_copy_client.cc:523] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:35.157054 5357 consensus_queue.cc:1035] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:35.157824 5357 consensus_queue.cc:1035] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:35.161083 5640 tablet_bootstrap.cc:492] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:35.162339 2202 raft_consensus.cc:1261] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:35.163165 5359 consensus_queue.cc:1035] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:35.197786 5658 raft_consensus.cc:2932] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:35.206092 4568 raft_consensus.cc:2932] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:35.208518 2072 consensus_queue.cc:237] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:35.207103 2202 raft_consensus.cc:2932] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:35.214498 1668 catalog_manager.cc:5484] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:35.215230 1943 raft_consensus.cc:2932] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:35.224804 2074 consensus_queue.cc:237] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:35.228473 4567 raft_consensus.cc:1261] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:35.229422 5359 consensus_queue.cc:1035] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:35.233000 5359 raft_consensus.cc:2932] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:35.234895 2202 raft_consensus.cc:1261] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:35.235179 2010 consensus_peers.cc:485] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a752938a2b6c424db0c08bb85b64429b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:35.235504 5573 consensus_queue.cc:1035] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:35.238447 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48f411ec0bfc4a318b661441c49bd6ec with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:35.239251 1944 raft_consensus.cc:1261] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:35.240711 1811 raft_consensus.cc:1261] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:35.240149 4567 raft_consensus.cc:2932] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:35.241482 5357 consensus_queue.cc:1035] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:35.241953 5357 consensus_queue.cc:1035] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:35.245044 1944 raft_consensus.cc:2932] T 48f411ec0bfc4a318b661441c49bd6ec P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:35.245872 5634 tablet_bootstrap.cc:492] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:35.246340 5634 tablet_bootstrap.cc:492] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:12:35.246650 5634 ts_tablet_manager.cc:1367] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.137s user 0.055s sys 0.011s
I20240503 08:12:35.249528 5634 raft_consensus.cc:348] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:35.249855 5634 raft_consensus.cc:729] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:12:35.249660 1663 catalog_manager.cc:5484] T 48f411ec0bfc4a318b661441c49bd6ec P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:35.250079 5634 consensus_queue.cc:260] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:35.251673 5634 ts_tablet_manager.cc:1398] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240503 08:12:35.252502 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-358cc7bd631d432385d6205db86cbfd4 received from {username='slave'} at 127.27.157.199:52311
I20240503 08:12:35.252676 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session f24648c92c384f1ba349eee801f985e2-358cc7bd631d432385d6205db86cbfd4 on tablet 358cc7bd631d432385d6205db86cbfd4 with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:12:35.254544 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f411ec0bfc4a318b661441c49bd6ec with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:35.255009 5535 tablet_replica.cc:330] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:35.255237 5535 raft_consensus.cc:2229] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:35.255533 5535 raft_consensus.cc:2258] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:35.255404 1668 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:35.256702 5535 ts_tablet_manager.cc:1875] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:35.270403 5357 consensus_queue.cc:1035] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:35.277511 5535 ts_tablet_manager.cc:1888] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:35.277786 5535 log.cc:1196] T 48f411ec0bfc4a318b661441c49bd6ec P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/48f411ec0bfc4a318b661441c49bd6ec
I20240503 08:12:35.278865 5190 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:35.280762 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 48f411ec0bfc4a318b661441c49bd6ec (table rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]) successfully deleted
I20240503 08:12:35.281991 1813 raft_consensus.cc:1261] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:35.282116 4703 raft_consensus.cc:1261] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:35.282908 5543 consensus_queue.cc:1035] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:35.283622 5650 consensus_queue.cc:1035] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:35.303730 5190 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:35.313696 1665 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c (table rebalancer_dynamic_table [id=52811fa0cad54e73a98995e835065069]). Expected version 1 got 0
I20240503 08:12:35.344400 1668 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c (table rebalancer_dynamic_table [id=52811fa0cad54e73a98995e835065069]). Expected version 1 got 0
I20240503 08:12:35.341706 5675 tablet.cc:1722] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:35.355021 5677 ts_tablet_manager.cc:902] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:35.355896 5676 tablet.cc:1722] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:35.358042 5675 tablet.cc:1620] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:35.357226 1668 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c (table rebalancer_dynamic_table [id=52811fa0cad54e73a98995e835065069]). Expected version 1 got 0
I20240503 08:12:35.355270 5640 tablet_bootstrap.cc:492] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:35.361863 5640 tablet_bootstrap.cc:492] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:35.362285 5640 ts_tablet_manager.cc:1367] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.201s user 0.056s sys 0.002s
I20240503 08:12:35.362418 5676 tablet.cc:1620] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:35.365639 5680 tablet.cc:1722] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:35.368674 5680 tablet.cc:1620] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:35.363878 5640 raft_consensus.cc:348] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:35.372560 5640 raft_consensus.cc:729] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:35.372905 5640 consensus_queue.cc:260] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:35.373261 5357 consensus_queue.cc:1035] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:35.373921 5640 ts_tablet_manager.cc:1398] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240503 08:12:35.376565 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-89d9b1b3c23241d49bfcb914a5bd1777 received from {username='slave'} at 127.27.157.197:56593
I20240503 08:12:35.376785 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-89d9b1b3c23241d49bfcb914a5bd1777 on tablet 89d9b1b3c23241d49bfcb914a5bd1777 with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:35.389386 1665 catalog_manager.cc:5576] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181) does not have the latest schema for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c (table rebalancer_dynamic_table [id=52811fa0cad54e73a98995e835065069]). Expected version 1 got 0
I20240503 08:12:35.391613 1663 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c (table rebalancer_dynamic_table [id=52811fa0cad54e73a98995e835065069]). Expected version 1 got 0
I20240503 08:12:35.393806 5658 raft_consensus.cc:2932] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:35.394629 2203 raft_consensus.cc:2932] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:35.403213 1665 catalog_manager.cc:5484] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:35.410629 4567 raft_consensus.cc:1203] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:35.413595 5417 consensus_queue.cc:1035] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:35.418548 1811 raft_consensus.cc:2932] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:35.432468 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=52811fa0cad54e73a98995e835065069] alter complete (version 1)
I20240503 08:12:35.449966 5417 consensus_queue.cc:1035] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:35.457464 5677 tablet_copy_client.cc:308] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.196:42183
I20240503 08:12:35.458854 2224 tablet_copy_service.cc:140] P 184c543cb05d49cc98ba1c56617be7a5: Received BeginTabletCopySession request for tablet 64973388b1c8466798b93ee424df489d from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:51901)
I20240503 08:12:35.459146 2224 tablet_copy_service.cc:161] P 184c543cb05d49cc98ba1c56617be7a5: Beginning new tablet copy session on tablet 64973388b1c8466798b93ee424df489d from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:51901: session id = 31f41542452b4b1eb994ff4223d039fc-64973388b1c8466798b93ee424df489d
I20240503 08:12:35.463095 2224 tablet_copy_source_session.cc:205] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:35.465011 5677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64973388b1c8466798b93ee424df489d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.470758 1811 consensus_queue.cc:237] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:35.471752 5677 tablet_copy_client.cc:783] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:35.472137 5677 tablet_copy_client.cc:647] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:35.474561 5677 tablet_copy_client.cc:523] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:35.475442 1943 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:35.476598 1672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:35.476282 5590 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:12:35.477488 1746 consensus_peers.cc:485] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3219159f92774adbb7ecaa77ac267021. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:35.478866 5677 tablet_bootstrap.cc:492] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:35.478914 2074 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:35.479593 5590 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:35.495549 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 3cda734f360c4d3eac88a6517028e49d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=a4efcdccd12c4091b793c47928fe4764]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:35.496098 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cda734f360c4d3eac88a6517028e49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.496843 5590 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:35.505074 1665 catalog_manager.cc:5484] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:35.507737 2074 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:35.525504 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 3cda734f360c4d3eac88a6517028e49d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=a4efcdccd12c4091b793c47928fe4764]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:35.526484 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cda734f360c4d3eac88a6517028e49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.528893 1923 tablet_service.cc:1467] Processing CreateTablet for tablet b572ab5c63cf4317a754987c5e50d221 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=a4efcdccd12c4091b793c47928fe4764]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:35.529405 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b572ab5c63cf4317a754987c5e50d221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.532378 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 3cda734f360c4d3eac88a6517028e49d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=a4efcdccd12c4091b793c47928fe4764]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:35.532846 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cda734f360c4d3eac88a6517028e49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.537564 5704 ts_tablet_manager.cc:902] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:35.545275 5710 tablet_bootstrap.cc:492] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:35.549449 4946 tablet_service.cc:1467] Processing CreateTablet for tablet b572ab5c63cf4317a754987c5e50d221 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=a4efcdccd12c4091b793c47928fe4764]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:35.549742 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 3e36553b05f0491dbd70a6d7eb12d7f5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=a4efcdccd12c4091b793c47928fe4764]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:35.550051 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b572ab5c63cf4317a754987c5e50d221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.550204 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e36553b05f0491dbd70a6d7eb12d7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.552242 2183 tablet_service.cc:1467] Processing CreateTablet for tablet b572ab5c63cf4317a754987c5e50d221 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=a4efcdccd12c4091b793c47928fe4764]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:35.552728 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b572ab5c63cf4317a754987c5e50d221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.553339 5710 tablet_bootstrap.cc:654] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:35.556941 4834 raft_consensus.cc:1203] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:35.560297 5710 tablet_bootstrap.cc:492] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:35.560587 5710 ts_tablet_manager.cc:1367] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20240503 08:12:35.561565 5710 raft_consensus.cc:348] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:35.561990 5710 raft_consensus.cc:374] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:35.562435 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 3e36553b05f0491dbd70a6d7eb12d7f5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=a4efcdccd12c4091b793c47928fe4764]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:35.563017 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e36553b05f0491dbd70a6d7eb12d7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.563387 5710 raft_consensus.cc:729] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:35.563972 5710 consensus_queue.cc:260] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:35.566092 5710 ts_tablet_manager.cc:1398] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:12:35.567955 5716 tablet_bootstrap.cc:492] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:35.569267 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 3e36553b05f0491dbd70a6d7eb12d7f5 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=a4efcdccd12c4091b793c47928fe4764]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:35.569752 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e36553b05f0491dbd70a6d7eb12d7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.572922 5716 tablet_bootstrap.cc:654] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:35.577095 1672 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:35.582790 1944 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:35.589138 5675 tablet.cc:1722] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:35.589686 5675 tablet.cc:1620] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:35.590287 5680 tablet.cc:1722] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:35.590795 5680 tablet.cc:1620] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:35.591841 5708 tablet_bootstrap.cc:492] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:35.592406 5716 tablet_bootstrap.cc:492] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:35.600011 5718 tablet_bootstrap.cc:492] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:35.601933 1663 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=52811fa0cad54e73a98995e835065069] alter complete (version 2)
I20240503 08:12:35.592911 5716 ts_tablet_manager.cc:1367] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20240503 08:12:35.606747 5717 tablet_bootstrap.cc:492] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:35.607527 5716 raft_consensus.cc:348] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:35.608492 5708 tablet_bootstrap.cc:654] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:35.608739 5716 raft_consensus.cc:374] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:35.609035 5716 raft_consensus.cc:729] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:35.609526 5716 consensus_queue.cc:260] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:35.612675 5714 tablet_bootstrap.cc:492] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:35.613369 5716 ts_tablet_manager.cc:1398] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 08:12:35.614586 5718 tablet_bootstrap.cc:654] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:35.615460 5634 ts_tablet_manager.cc:902] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:35.616127 5717 tablet_bootstrap.cc:654] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:35.629086 5714 tablet_bootstrap.cc:654] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:35.637043 5714 tablet_bootstrap.cc:492] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:35.637949 5714 ts_tablet_manager.cc:1367] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240503 08:12:35.638831 5676 tablet.cc:1722] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:35.639134 5714 raft_consensus.cc:348] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:35.640771 5714 raft_consensus.cc:374] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:35.641088 5676 tablet.cc:1620] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:35.641250 5714 raft_consensus.cc:729] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:35.641772 5714 consensus_queue.cc:260] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:35.643898 5714 ts_tablet_manager.cc:1398] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:35.644796 5714 tablet_bootstrap.cc:492] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:35.646693 1944 consensus_queue.cc:237] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:35.648733 5714 tablet_bootstrap.cc:654] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:35.650431 5718 tablet_bootstrap.cc:492] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:35.650717 5718 ts_tablet_manager.cc:1367] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.005s
I20240503 08:12:35.651798 5708 tablet_bootstrap.cc:492] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:35.652966 5708 ts_tablet_manager.cc:1367] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.001s
I20240503 08:12:35.654167 5708 raft_consensus.cc:348] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:35.654868 5718 raft_consensus.cc:348] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:35.655143 5718 raft_consensus.cc:374] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:35.655242 5718 raft_consensus.cc:729] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:35.655685 5708 raft_consensus.cc:374] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:35.655629 5718 consensus_queue.cc:260] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:35.655949 5708 raft_consensus.cc:729] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:35.656526 5708 consensus_queue.cc:260] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:35.659420 5718 ts_tablet_manager.cc:1398] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 08:12:35.660070 5717 tablet_bootstrap.cc:492] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:35.660380 5717 ts_tablet_manager.cc:1367] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.008s
I20240503 08:12:35.661343 5717 raft_consensus.cc:348] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:35.661725 5717 raft_consensus.cc:374] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:35.661870 5717 raft_consensus.cc:729] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:35.662858 5717 consensus_queue.cc:260] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:35.663517 5708 ts_tablet_manager.cc:1398] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 08:12:35.664494 5717 ts_tablet_manager.cc:1398] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:35.664995 5708 tablet_bootstrap.cc:492] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:35.667905 5729 tablet_bootstrap.cc:492] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:35.669780 5729 tablet_bootstrap.cc:654] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:35.671825 5708 tablet_bootstrap.cc:654] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:35.672701 5714 tablet_bootstrap.cc:492] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:35.672936 5714 ts_tablet_manager.cc:1367] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.004s
I20240503 08:12:35.673803 5714 raft_consensus.cc:348] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:35.674139 5714 raft_consensus.cc:374] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:35.674278 5714 raft_consensus.cc:729] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:35.674628 5714 consensus_queue.cc:260] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:35.674818 5704 tablet_copy_client.cc:308] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:12:35.676245 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet 820ecfef09a34327a08f7c3a893ccb05 from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:54589)
I20240503 08:12:35.676445 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet 820ecfef09a34327a08f7c3a893ccb05 from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:54589: session id = b8f581390da449ccb9e299011296405e-820ecfef09a34327a08f7c3a893ccb05
I20240503 08:12:35.682389 5634 tablet_copy_client.cc:308] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:35.683287 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet a752938a2b6c424db0c08bb85b64429b from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:52311)
I20240503 08:12:35.683542 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet a752938a2b6c424db0c08bb85b64429b from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:52311: session id = f24648c92c384f1ba349eee801f985e2-a752938a2b6c424db0c08bb85b64429b
I20240503 08:12:35.684257 5708 tablet_bootstrap.cc:492] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:35.685062 4855 tablet_copy_source_session.cc:205] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 08:12:35.685598 1880 consensus_peers.cc:485] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c37418ac97343d1931a8e1239218a38. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:35.686640 5714 ts_tablet_manager.cc:1398] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240503 08:12:35.686687 2203 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:35.687350 5417 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:35.688491 5704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820ecfef09a34327a08f7c3a893ccb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.691996 2074 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:35.692716 5417 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:35.696952 2093 tablet_copy_source_session.cc:205] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:35.698938 5634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a752938a2b6c424db0c08bb85b64429b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.702454 5729 tablet_bootstrap.cc:492] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:35.702723 5729 ts_tablet_manager.cc:1367] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.004s
I20240503 08:12:35.703646 5729 raft_consensus.cc:348] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:35.703941 5729 raft_consensus.cc:374] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:35.704082 5729 raft_consensus.cc:729] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:35.704401 5729 consensus_queue.cc:260] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:35.706068 5729 ts_tablet_manager.cc:1398] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:35.707859 5708 ts_tablet_manager.cc:1367] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.003s
I20240503 08:12:35.707921 1663 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:35.708878 5708 raft_consensus.cc:348] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:35.710625 5533 raft_consensus.cc:482] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:35.710809 5533 raft_consensus.cc:504] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:35.711696 5533 leader_election.cc:288] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:35.712751 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572ab5c63cf4317a754987c5e50d221" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:35.712985 2202 raft_consensus.cc:2454] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:35.713780 4901 leader_election.cc:302] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 184c543cb05d49cc98ba1c56617be7a5; no voters:
I20240503 08:12:35.714392 5533 raft_consensus.cc:2781] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:35.714514 5533 raft_consensus.cc:482] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:35.714627 5533 raft_consensus.cc:3037] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.718097 5533 raft_consensus.cc:504] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:35.719746 5675 tablet.cc:1722] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:35.720656 5676 tablet.cc:1722] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:35.727303 5634 tablet_copy_client.cc:783] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:35.727687 5634 tablet_copy_client.cc:647] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:35.728068 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572ab5c63cf4317a754987c5e50d221" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:35.728322 1942 raft_consensus.cc:3037] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.729557 5677 tablet_bootstrap.cc:492] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:35.729558 5634 tablet_copy_client.cc:523] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:35.729965 5677 tablet_bootstrap.cc:492] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:12:35.730299 5677 ts_tablet_manager.cc:1367] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.252s user 0.069s sys 0.082s
I20240503 08:12:35.731175 1942 raft_consensus.cc:2454] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:35.731834 5677 raft_consensus.cc:348] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:35.732103 5677 raft_consensus.cc:729] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:12:35.732321 5677 consensus_queue.cc:260] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:35.732931 5677 ts_tablet_manager.cc:1398] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 08:12:35.734025 2224 tablet_copy_service.cc:342] P 184c543cb05d49cc98ba1c56617be7a5: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-64973388b1c8466798b93ee424df489d received from {username='slave'} at 127.27.157.198:51901
I20240503 08:12:35.734227 2224 tablet_copy_service.cc:433] P 184c543cb05d49cc98ba1c56617be7a5: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-64973388b1c8466798b93ee424df489d on tablet 64973388b1c8466798b93ee424df489d with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:35.735605 4902 leader_election.cc:302] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:35.736591 5737 raft_consensus.cc:2781] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:35.737639 5533 leader_election.cc:288] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:35.737676 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572ab5c63cf4317a754987c5e50d221" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:35.737949 2201 raft_consensus.cc:3037] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.738148 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572ab5c63cf4317a754987c5e50d221" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:35.738461 1944 raft_consensus.cc:2362] T b572ab5c63cf4317a754987c5e50d221 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1. Re-sending same reply.
I20240503 08:12:35.740146 5737 raft_consensus.cc:686] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:35.740473 5737 consensus_queue.cc:237] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:35.740702 2201 raft_consensus.cc:2454] T b572ab5c63cf4317a754987c5e50d221 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:35.742250 5553 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:35.742801 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=52811fa0cad54e73a98995e835065069] alter complete (version 3)
I20240503 08:12:35.744294 2201 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:35.750260 5634 tablet_bootstrap.cc:492] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:35.753568 1665 catalog_manager.cc:5484] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:35.712869 5704 tablet_copy_client.cc:783] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:35.759533 5708 raft_consensus.cc:374] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:35.759780 5704 tablet_copy_client.cc:647] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:35.733284 5680 tablet.cc:1722] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:35.762611 5704 tablet_copy_client.cc:523] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:35.766062 5708 raft_consensus.cc:729] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:35.766633 5708 consensus_queue.cc:260] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:35.766803 5704 tablet_bootstrap.cc:492] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:35.776564 5708 ts_tablet_manager.cc:1398] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.069s user 0.001s sys 0.001s
I20240503 08:12:35.783627 2074 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:35.809919 1665 catalog_manager.cc:5484] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:35.814409 5650 raft_consensus.cc:482] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:35.814615 5650 raft_consensus.cc:504] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:35.815366 5650 leader_election.cc:288] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:35.815833 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cda734f360c4d3eac88a6517028e49d" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:35.816071 1811 raft_consensus.cc:2454] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:35.816200 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cda734f360c4d3eac88a6517028e49d" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:35.816463 4834 raft_consensus.cc:2454] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:35.816499 4503 leader_election.cc:302] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:35.816815 5650 raft_consensus.cc:2781] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:35.816937 5650 raft_consensus.cc:482] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:35.817047 5650 raft_consensus.cc:3037] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.820500 5704 tablet_bootstrap.cc:492] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:35.821095 5704 tablet_bootstrap.cc:492] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:12:35.821401 5704 ts_tablet_manager.cc:1367] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.007s
I20240503 08:12:35.821969 5650 raft_consensus.cc:504] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:35.822482 5704 raft_consensus.cc:348] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:35.822777 5704 raft_consensus.cc:729] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:12:35.823032 5650 leader_election.cc:288] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:35.823089 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cda734f360c4d3eac88a6517028e49d" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:35.823305 1811 raft_consensus.cc:3037] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.823591 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cda734f360c4d3eac88a6517028e49d" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:35.823846 4834 raft_consensus.cc:3037] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.826390 1811 raft_consensus.cc:2454] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:35.826587 4834 raft_consensus.cc:2454] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:35.826933 4503 leader_election.cc:302] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:35.827713 5650 raft_consensus.cc:2781] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:35.827944 5650 raft_consensus.cc:686] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:35.828341 5650 consensus_queue.cc:237] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:35.837157 5704 consensus_queue.cc:260] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:35.838456 5704 ts_tablet_manager.cc:1398] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240503 08:12:35.845109 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session b8f581390da449ccb9e299011296405e-820ecfef09a34327a08f7c3a893ccb05 received from {username='slave'} at 127.27.157.193:54589
I20240503 08:12:35.845338 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session b8f581390da449ccb9e299011296405e-820ecfef09a34327a08f7c3a893ccb05 on tablet 820ecfef09a34327a08f7c3a893ccb05 with peer b8f581390da449ccb9e299011296405e
I20240503 08:12:35.846799 1663 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:35.847029 1663 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:35.857630 1663 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 01f7c4ff8fae42c4bedaa95a16e6f07c
I20240503 08:12:35.858619 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:35 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:35.858597 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:35 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:35.869117 1665 catalog_manager.cc:5484] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:35.869980 5750 tablet_replica.cc:330] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:35.873960 5751 tablet_replica.cc:330] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:35.874460 5751 raft_consensus.cc:2229] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:35.874981 5751 raft_consensus.cc:2258] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:35.870244 5750 raft_consensus.cc:2229] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:35.877003 5750 raft_consensus.cc:2258] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:35.878754 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f7c4ff8fae42c4bedaa95a16e6f07c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:35 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:35.879447 5752 tablet_replica.cc:330] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:35.879552 5751 ts_tablet_manager.cc:1875] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:35.879734 5752 raft_consensus.cc:2229] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:35.880175 5752 raft_consensus.cc:2258] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:35.880170 5750 ts_tablet_manager.cc:1875] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:35.881196 5752 ts_tablet_manager.cc:1875] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:35.889158 5752 ts_tablet_manager.cc:1888] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:35.889417 5752 log.cc:1196] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/01f7c4ff8fae42c4bedaa95a16e6f07c
I20240503 08:12:35.890084 5752 ts_tablet_manager.cc:1909] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:35.891602 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 01f7c4ff8fae42c4bedaa95a16e6f07c (table rebalancer_dynamic_table_new_name [id=52811fa0cad54e73a98995e835065069]) successfully deleted
I20240503 08:12:35.896703 5750 ts_tablet_manager.cc:1888] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:35.900981 5750 log.cc:1196] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/01f7c4ff8fae42c4bedaa95a16e6f07c
I20240503 08:12:35.901885 5751 ts_tablet_manager.cc:1888] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:35.902112 5751 log.cc:1196] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/01f7c4ff8fae42c4bedaa95a16e6f07c
I20240503 08:12:35.902268 4703 raft_consensus.cc:1203] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:35.902693 5751 ts_tablet_manager.cc:1909] T 01f7c4ff8fae42c4bedaa95a16e6f07c P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:35.906966 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 01f7c4ff8fae42c4bedaa95a16e6f07c (table rebalancer_dynamic_table_new_name [id=52811fa0cad54e73a98995e835065069]) successfully deleted
I20240503 08:12:35.914944 5531 raft_consensus.cc:482] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:35.923929 5531 raft_consensus.cc:504] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:35.924453 5750 ts_tablet_manager.cc:1909] T 01f7c4ff8fae42c4bedaa95a16e6f07c P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:35.924849 5634 tablet_bootstrap.cc:492] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:35.924892 5531 leader_election.cc:288] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:35.925329 5634 tablet_bootstrap.cc:492] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:12:35.925624 5634 ts_tablet_manager.cc:1367] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.175s user 0.056s sys 0.003s
I20240503 08:12:35.925926 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e36553b05f0491dbd70a6d7eb12d7f5" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:35.926170 4834 raft_consensus.cc:2454] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:35.926566 1749 leader_election.cc:302] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f581390da449ccb9e299011296405e, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:35.926529 5634 raft_consensus.cc:348] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:35.926788 5634 raft_consensus.cc:729] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:12:35.926981 5531 raft_consensus.cc:2781] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:35.927089 5531 raft_consensus.cc:482] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:35.927196 5531 raft_consensus.cc:3037] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.927052 5634 consensus_queue.cc:260] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:35.927690 5634 ts_tablet_manager.cc:1398] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:35.928989 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-a752938a2b6c424db0c08bb85b64429b received from {username='slave'} at 127.27.157.199:52311
I20240503 08:12:35.929157 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session f24648c92c384f1ba349eee801f985e2-a752938a2b6c424db0c08bb85b64429b on tablet a752938a2b6c424db0c08bb85b64429b with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:12:35.931121 5531 raft_consensus.cc:504] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:35.932544 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e36553b05f0491dbd70a6d7eb12d7f5" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:35.932770 4834 raft_consensus.cc:3037] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.933557 5531 leader_election.cc:288] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:35.936590 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 01f7c4ff8fae42c4bedaa95a16e6f07c (table rebalancer_dynamic_table_new_name [id=52811fa0cad54e73a98995e835065069]) successfully deleted
I20240503 08:12:35.936851 4834 raft_consensus.cc:2454] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:35.937732 1749 leader_election.cc:302] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f581390da449ccb9e299011296405e, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:35.938092 5531 raft_consensus.cc:2781] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:35.938268 5531 raft_consensus.cc:686] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:35.938557 5531 consensus_queue.cc:237] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:35.944404 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e36553b05f0491dbd70a6d7eb12d7f5" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:35.944680 4703 raft_consensus.cc:2454] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:35.944996 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e36553b05f0491dbd70a6d7eb12d7f5" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:35.945164 4702 raft_consensus.cc:3037] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:35.946813 1663 catalog_manager.cc:5484] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:35.948892 4702 raft_consensus.cc:2454] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:35.979880 1663 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:35.989570 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 0e912869a85b4f708d2f3d2562920649 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e732af17bba7429da9c7c8b3e8d4a0a2]), partition=
I20240503 08:12:35.989697 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 0e912869a85b4f708d2f3d2562920649 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e732af17bba7429da9c7c8b3e8d4a0a2]), partition=
I20240503 08:12:35.990024 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e912869a85b4f708d2f3d2562920649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.990065 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e912869a85b4f708d2f3d2562920649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:35.993602 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 0e912869a85b4f708d2f3d2562920649 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e732af17bba7429da9c7c8b3e8d4a0a2]), partition=
I20240503 08:12:35.993973 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e912869a85b4f708d2f3d2562920649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.000810 5708 tablet_bootstrap.cc:492] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:36.002169 5708 tablet_bootstrap.cc:654] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.010021 5573 raft_consensus.cc:1050] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:36.010907 5573 consensus_queue.cc:237] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:36.021163 5714 tablet_bootstrap.cc:492] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:36.022785 5714 tablet_bootstrap.cc:654] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.025552 5729 tablet_bootstrap.cc:492] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:36.027428 5729 tablet_bootstrap.cc:654] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.027416 4567 raft_consensus.cc:1261] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.028208 5357 consensus_queue.cc:1035] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:36.034658 2201 raft_consensus.cc:1261] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.034920 1942 raft_consensus.cc:1261] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.035532 5573 consensus_queue.cc:1035] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:36.037159 5357 consensus_queue.cc:1035] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:36.037447 5708 tablet_bootstrap.cc:492] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:36.037603 5708 ts_tablet_manager.cc:1367] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.001s
I20240503 08:12:36.038512 5708 raft_consensus.cc:348] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.038720 5708 raft_consensus.cc:374] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.038930 5708 raft_consensus.cc:729] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.039302 5708 consensus_queue.cc:260] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.040488 5708 ts_tablet_manager.cc:1398] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 08:12:36.070423 5729 tablet_bootstrap.cc:492] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:36.071113 5729 ts_tablet_manager.cc:1367] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.005s
I20240503 08:12:36.076320 4834 raft_consensus.cc:1203] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:36.078722 5658 raft_consensus.cc:2932] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:36.084133 4568 raft_consensus.cc:2932] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:36.088104 1942 raft_consensus.cc:2932] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:36.073809 5729 raft_consensus.cc:348] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.095067 5729 raft_consensus.cc:374] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.095180 5729 raft_consensus.cc:729] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.096076 5729 consensus_queue.cc:260] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.096685 5729 ts_tablet_manager.cc:1398] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240503 08:12:36.096530 2201 raft_consensus.cc:2932] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:36.098742 5785 ts_tablet_manager.cc:902] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:36.101089 1812 raft_consensus.cc:1203] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:36.107092 5531 raft_consensus.cc:482] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:36.107273 5531 raft_consensus.cc:504] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.108062 5531 leader_election.cc:288] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:36.107087 1670 catalog_manager.cc:5484] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:36.108850 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e912869a85b4f708d2f3d2562920649" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:36.109115 4702 raft_consensus.cc:2454] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:36.109510 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e912869a85b4f708d2f3d2562920649" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:36.109653 1749 leader_election.cc:302] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:36.110703 5407 raft_consensus.cc:2781] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:36.110823 5407 raft_consensus.cc:482] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:36.110965 5407 raft_consensus.cc:3037] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:12:36.112195 1749 leader_election.cc:341] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:36.123859 5714 tablet_bootstrap.cc:492] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:36.124050 5714 ts_tablet_manager.cc:1367] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.009s
I20240503 08:12:36.124958 5714 raft_consensus.cc:348] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.125190 5714 raft_consensus.cc:374] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.125280 5714 raft_consensus.cc:729] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.125558 5714 consensus_queue.cc:260] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.127415 5714 ts_tablet_manager.cc:1398] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:36.133275 2074 consensus_queue.cc:237] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:36.115248 5407 raft_consensus.cc:504] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.134491 5407 leader_election.cc:288] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:36.135206 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e912869a85b4f708d2f3d2562920649" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:36.135552 4702 raft_consensus.cc:3037] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.135531 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e912869a85b4f708d2f3d2562920649" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:36.135773 4834 raft_consensus.cc:3037] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.138569 4702 raft_consensus.cc:2454] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:36.138579 4834 raft_consensus.cc:2454] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:36.141796 1749 leader_election.cc:302] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:36.150283 4568 raft_consensus.cc:1261] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:36.150269 2201 raft_consensus.cc:1261] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:36.151041 5658 consensus_queue.cc:1035] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:36.151947 5573 consensus_queue.cc:1035] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:36.152827 5573 raft_consensus.cc:1050] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:12:36.153698 5573 consensus_queue.cc:237] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:36.156772 5407 raft_consensus.cc:2781] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:36.157497 5407 raft_consensus.cc:686] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:36.157927 5407 consensus_queue.cc:237] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.158453 5784 raft_consensus.cc:2932] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:36.165154 4568 raft_consensus.cc:2932] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:36.175848 2201 raft_consensus.cc:2932] T 89d9b1b3c23241d49bfcb914a5bd1777 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:36.179286 1670 catalog_manager.cc:5484] T 89d9b1b3c23241d49bfcb914a5bd1777 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:36.184715 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 89d9b1b3c23241d49bfcb914a5bd1777 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:36.188252 1672 catalog_manager.cc:5484] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:36.190147 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 89d9b1b3c23241d49bfcb914a5bd1777 with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:36.191306 5794 tablet_replica.cc:330] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:36.191642 5794 raft_consensus.cc:2229] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:36.192178 5794 raft_consensus.cc:2258] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:36.203055 5794 ts_tablet_manager.cc:1875] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:36.211426 2201 raft_consensus.cc:1261] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.212095 5784 consensus_queue.cc:1035] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:36.212234 4834 raft_consensus.cc:1261] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.216416 5573 consensus_queue.cc:1035] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:36.224552 1942 raft_consensus.cc:1261] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.230963 5573 consensus_queue.cc:1035] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:36.231779 5794 ts_tablet_manager.cc:1888] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:36.231935 5794 log.cc:1196] T 89d9b1b3c23241d49bfcb914a5bd1777 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/89d9b1b3c23241d49bfcb914a5bd1777
I20240503 08:12:36.258739 5573 raft_consensus.cc:2932] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.263008 5785 tablet_copy_client.cc:308] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:41181
I20240503 08:12:36.263078 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 89d9b1b3c23241d49bfcb914a5bd1777 (table rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]) successfully deleted
I20240503 08:12:36.264307 1833 tablet_copy_service.cc:140] P b8f581390da449ccb9e299011296405e: Received BeginTabletCopySession request for tablet 3219159f92774adbb7ecaa77ac267021 from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:43085)
I20240503 08:12:36.264528 1833 tablet_copy_service.cc:161] P b8f581390da449ccb9e299011296405e: Beginning new tablet copy session on tablet 3219159f92774adbb7ecaa77ac267021 from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:43085: session id = 5f95b05d111e4090bbc59dec394ed0ab-3219159f92774adbb7ecaa77ac267021
I20240503 08:12:36.266135 1942 raft_consensus.cc:2932] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.268214 2201 raft_consensus.cc:2932] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.269199 1833 tablet_copy_source_session.cc:205] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:36.270033 1665 catalog_manager.cc:5484] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:36.279296 5737 consensus_queue.cc:1035] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:36.280180 2074 consensus_queue.cc:237] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:36.284421 1944 raft_consensus.cc:1261] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:36.285377 5357 consensus_queue.cc:1035] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:36.265743 4834 raft_consensus.cc:2932] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.286607 5785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3219159f92774adbb7ecaa77ac267021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.292264 5650 consensus_queue.cc:1035] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:36.297667 5784 raft_consensus.cc:2932] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.298369 1944 raft_consensus.cc:2932] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.302769 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 358cc7bd631d432385d6205db86cbfd4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:36.308269 5756 raft_consensus.cc:1050] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:12:36.307813 1665 catalog_manager.cc:5484] T 358cc7bd631d432385d6205db86cbfd4 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.309516 5756 consensus_queue.cc:237] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:36.314882 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 358cc7bd631d432385d6205db86cbfd4 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:36.317171 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:36.334722 5802 ts_tablet_manager.cc:902] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:36.337831 5809 tablet_replica.cc:330] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:36.338090 5809 raft_consensus.cc:2229] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:36.338470 5809 raft_consensus.cc:2258] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:36.340211 4835 raft_consensus.cc:1261] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:36.341861 5809 ts_tablet_manager.cc:1875] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:36.343820 5784 consensus_queue.cc:1035] T 358cc7bd631d432385d6205db86cbfd4 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:36.345947 4835 raft_consensus.cc:2932] T 358cc7bd631d432385d6205db86cbfd4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.357388 5809 ts_tablet_manager.cc:1888] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:36.359490 5809 log.cc:1196] T 358cc7bd631d432385d6205db86cbfd4 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/358cc7bd631d432385d6205db86cbfd4
I20240503 08:12:36.361891 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 358cc7bd631d432385d6205db86cbfd4 (table rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]) successfully deleted
I20240503 08:12:36.372026 5737 consensus_queue.cc:1035] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:36.383375 5531 consensus_queue.cc:1035] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:36.397861 4702 raft_consensus.cc:1261] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.398563 2074 raft_consensus.cc:1261] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.399665 5755 consensus_queue.cc:1035] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:36.400355 5755 consensus_queue.cc:1035] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:36.401208 1944 raft_consensus.cc:1261] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.404230 5531 consensus_queue.cc:1035] T 3e36553b05f0491dbd70a6d7eb12d7f5 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:36.405767 5753 consensus_queue.cc:1035] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:36.437944 5785 tablet_copy_client.cc:783] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:36.438542 5785 tablet_copy_client.cc:647] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:36.442029 5785 tablet_copy_client.cc:523] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:36.453644 5785 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:36.454995 5650 consensus_queue.cc:1035] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:36.465839 4703 raft_consensus.cc:1261] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:36.467005 4833 raft_consensus.cc:1261] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:36.467648 5814 consensus_queue.cc:1035] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 08:12:36.470149 5755 raft_consensus.cc:2932] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:36.470710 5531 consensus_queue.cc:1035] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:36.476227 2072 raft_consensus.cc:2932] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:36.489264 1663 catalog_manager.cc:5484] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:36.508636 5840 tablet.cc:1722] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:36.509255 5840 tablet.cc:1620] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:36.524267 5843 tablet.cc:1722] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:36.525981 1670 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:36.527113 1663 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 0e912869a85b4f708d2f3d2562920649 (table rebalancer_dynamic_table [id=e732af17bba7429da9c7c8b3e8d4a0a2]). Expected version 1 got 0
I20240503 08:12:36.528120 5843 tablet.cc:1620] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:36.534700 5357 raft_consensus.cc:1050] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:12:36.536295 5357 consensus_queue.cc:237] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:36.548758 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 1e641fe703c14f4cb1bd22c676d79576 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=b06077fc3c724567becc26a279a4b8ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.549269 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e641fe703c14f4cb1bd22c676d79576. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.549614 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 1e641fe703c14f4cb1bd22c676d79576 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=b06077fc3c724567becc26a279a4b8ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.550058 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e641fe703c14f4cb1bd22c676d79576. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.551745 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 0caddf5e39f8440baa8ad3656ce91a8e (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=b06077fc3c724567becc26a279a4b8ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.552121 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caddf5e39f8440baa8ad3656ce91a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.556545 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 0caddf5e39f8440baa8ad3656ce91a8e (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=b06077fc3c724567becc26a279a4b8ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.557070 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caddf5e39f8440baa8ad3656ce91a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.562244 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 0caddf5e39f8440baa8ad3656ce91a8e (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=b06077fc3c724567becc26a279a4b8ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.562739 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caddf5e39f8440baa8ad3656ce91a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.567523 5729 tablet_bootstrap.cc:492] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:36.568913 4947 tablet_service.cc:1467] Processing CreateTablet for tablet 296de1897a5e4969aa36b501080a35d9 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=b06077fc3c724567becc26a279a4b8ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.569392 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296de1897a5e4969aa36b501080a35d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.570012 5729 tablet_bootstrap.cc:654] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.584254 1663 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=e732af17bba7429da9c7c8b3e8d4a0a2] alter complete (version 1)
I20240503 08:12:36.584404 5856 tablet_bootstrap.cc:492] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:36.587600 5856 tablet_bootstrap.cc:654] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.589042 5854 tablet_bootstrap.cc:492] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:36.589429 5714 tablet_bootstrap.cc:492] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:36.592106 5714 tablet_bootstrap.cc:654] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.592761 5854 tablet_bootstrap.cc:654] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.605804 4682 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:12:36.606717 5729 tablet_bootstrap.cc:492] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:36.606906 5729 ts_tablet_manager.cc:1367] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.002s
I20240503 08:12:36.608048 5729 raft_consensus.cc:348] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:36.614127 5729 raft_consensus.cc:374] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.614331 5729 raft_consensus.cc:729] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.614799 5729 consensus_queue.cc:260] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:36.615625 5729 ts_tablet_manager.cc:1398] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240503 08:12:36.629346 5854 tablet_bootstrap.cc:492] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:36.629824 5854 ts_tablet_manager.cc:1367] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20240503 08:12:36.632180 5854 raft_consensus.cc:348] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:36.633440 5854 raft_consensus.cc:374] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.633752 5854 raft_consensus.cc:729] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.634214 5854 consensus_queue.cc:260] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:36.634523 5856 tablet_bootstrap.cc:492] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:36.634810 5856 ts_tablet_manager.cc:1367] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.003s
I20240503 08:12:36.635566 5854 ts_tablet_manager.cc:1398] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:36.636512 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 296de1897a5e4969aa36b501080a35d9 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=b06077fc3c724567becc26a279a4b8ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.636992 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296de1897a5e4969aa36b501080a35d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.641975 5854 tablet_bootstrap.cc:492] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:36.643754 1663 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet 0e912869a85b4f708d2f3d2562920649 (table rebalancer_dynamic_table [id=e732af17bba7429da9c7c8b3e8d4a0a2]). Expected version 1 got 0
I20240503 08:12:36.643862 5854 tablet_bootstrap.cc:654] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.657215 1944 raft_consensus.cc:2932] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:36.670056 4702 raft_consensus.cc:2932] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:36.671224 2201 consensus_queue.cc:237] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:36.674556 1812 raft_consensus.cc:1261] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.674587 2202 raft_consensus.cc:1261] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.674751 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 1e641fe703c14f4cb1bd22c676d79576 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=b06077fc3c724567becc26a279a4b8ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.675289 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e641fe703c14f4cb1bd22c676d79576. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.675313 5784 consensus_queue.cc:1035] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:36.676084 5784 consensus_queue.cc:1035] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:36.679775 4547 tablet_service.cc:1467] Processing CreateTablet for tablet 296de1897a5e4969aa36b501080a35d9 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=b06077fc3c724567becc26a279a4b8ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:36.680346 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296de1897a5e4969aa36b501080a35d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.689639 5714 tablet_bootstrap.cc:492] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:36.689946 5714 ts_tablet_manager.cc:1367] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.101s user 0.007s sys 0.000s
I20240503 08:12:36.691011 5714 raft_consensus.cc:348] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.691677 5714 raft_consensus.cc:374] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.691826 5714 raft_consensus.cc:729] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.692189 5714 consensus_queue.cc:260] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.693300 5714 ts_tablet_manager.cc:1398] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:36.701993 4702 raft_consensus.cc:1261] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:36.703931 5786 raft_consensus.cc:1050] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:12:36.705227 5786 consensus_queue.cc:237] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:12:36.706838 5868 tablet_bootstrap.cc:492] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:36.709589 5868 tablet_bootstrap.cc:654] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.715559 4833 raft_consensus.cc:1261] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:36.718256 5856 raft_consensus.cc:348] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:36.718681 5856 raft_consensus.cc:374] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.718863 5856 raft_consensus.cc:729] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.719192 5856 consensus_queue.cc:260] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:36.719936 4947 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:12:36.721309 5856 ts_tablet_manager.cc:1398] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.086s user 0.002s sys 0.000s
I20240503 08:12:36.721737 5856 tablet_bootstrap.cc:492] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:36.722743 1670 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:36.725914 4967 raft_consensus.cc:1261] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:36.726725 5190 consensus_queue.cc:1035] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:36.727699 1810 raft_consensus.cc:1261] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:36.728413 5190 consensus_queue.cc:1035] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:36.735765 5802 tablet_copy_client.cc:308] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:36.735834 5357 consensus_queue.cc:1035] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:12:36.736943 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 7c37418ac97343d1931a8e1239218a38 from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:44359)
I20240503 08:12:36.737263 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 7c37418ac97343d1931a8e1239218a38 from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:44359: session id = 31f41542452b4b1eb994ff4223d039fc-7c37418ac97343d1931a8e1239218a38
I20240503 08:12:36.738870 5869 tablet_bootstrap.cc:492] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:36.741932 5869 tablet_bootstrap.cc:654] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.743515 1964 tablet_copy_source_session.cc:205] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:36.744238 5865 tablet.cc:1722] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:36.748067 5865 tablet.cc:1620] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:36.747761 5573 raft_consensus.cc:2932] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.754968 5840 tablet.cc:1722] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:36.757170 5840 tablet.cc:1620] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:36.763906 5843 tablet.cc:1722] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:36.764444 5843 tablet.cc:1620] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:36.765323 4812 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 08:12:36.770153 5865 tablet.cc:1722] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:36.771497 5865 tablet.cc:1620] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:36.773531 5856 tablet_bootstrap.cc:654] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:36.773663 1812 raft_consensus.cc:2932] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.777271 5802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c37418ac97343d1931a8e1239218a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:36.793721 1663 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet 0e912869a85b4f708d2f3d2562920649 (table rebalancer_dynamic_table [id=e732af17bba7429da9c7c8b3e8d4a0a2]). Expected version 2 got 0
I20240503 08:12:36.794703 1670 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=e732af17bba7429da9c7c8b3e8d4a0a2] alter complete (version 2)
I20240503 08:12:36.804819 4835 raft_consensus.cc:2932] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.805836 5305 raft_consensus.cc:482] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:36.806030 5305 raft_consensus.cc:504] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:36.806913 5305 leader_election.cc:288] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:36.807718 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caddf5e39f8440baa8ad3656ce91a8e" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
W20240503 08:12:36.808851 4636 leader_election.cc:341] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:36.808885 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caddf5e39f8440baa8ad3656ce91a8e" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:36.809172 1811 raft_consensus.cc:2454] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:36.809703 4637 leader_election.cc:302] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters: 00b61625d8a7460892c118a2570afda5
I20240503 08:12:36.810293 5305 raft_consensus.cc:2781] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:36.810420 5305 raft_consensus.cc:482] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:36.810510 5305 raft_consensus.cc:3037] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.813315 5305 raft_consensus.cc:504] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:36.815263 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caddf5e39f8440baa8ad3656ce91a8e" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:36.815701 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caddf5e39f8440baa8ad3656ce91a8e" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
W20240503 08:12:36.815928 4636 leader_election.cc:341] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:36.815999 1811 raft_consensus.cc:3037] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:36.818296 1943 raft_consensus.cc:1261] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:36.821668 1670 catalog_manager.cc:5484] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.823653 1923 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:12:36.834513 2054 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:12:36.842595 5305 leader_election.cc:288] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:36.852599 2074 consensus_queue.cc:237] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:36.855197 1811 raft_consensus.cc:2454] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:36.857249 4835 raft_consensus.cc:1261] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:36.857851 4637 leader_election.cc:302] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters: 00b61625d8a7460892c118a2570afda5
I20240503 08:12:36.858165 1812 raft_consensus.cc:1261] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:36.858232 5357 consensus_queue.cc:1035] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:36.858553 5305 raft_consensus.cc:2781] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:36.715498 5755 consensus_queue.cc:1035] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:36.858850 5784 consensus_queue.cc:1035] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:36.859427 5753 consensus_queue.cc:1035] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:36.763890 2201 raft_consensus.cc:2932] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.861290 5784 raft_consensus.cc:2932] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.861330 5786 raft_consensus.cc:2932] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:36.775890 2202 raft_consensus.cc:1261] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:36.863512 5190 consensus_queue.cc:1035] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:36.865731 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a752938a2b6c424db0c08bb85b64429b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:36.866708 4835 raft_consensus.cc:2932] T a752938a2b6c424db0c08bb85b64429b P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.866842 5753 raft_consensus.cc:2932] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:36.870914 4702 raft_consensus.cc:2932] T 64973388b1c8466798b93ee424df489d P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:36.874910 1670 catalog_manager.cc:5484] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:36.875466 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64973388b1c8466798b93ee424df489d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:36.875445 1810 raft_consensus.cc:2932] T a752938a2b6c424db0c08bb85b64429b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:36.877214 1663 catalog_manager.cc:5484] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:36.881352 2183 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 08:12:36.883431 4967 raft_consensus.cc:2932] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:36.889783 1811 raft_consensus.cc:2932] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:36.893946 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 64973388b1c8466798b93ee424df489d with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:36.896222 1943 raft_consensus.cc:2932] T 64973388b1c8466798b93ee424df489d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:36.899703 1671 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:36.900743 1668 catalog_manager.cc:5484] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:36.904078 1791 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:12:36.909449 4548 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:12:36.917279 5840 tablet.cc:1722] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:36.926158 5854 tablet_bootstrap.cc:492] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:36.926735 5854 ts_tablet_manager.cc:1367] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.285s user 0.013s sys 0.037s
I20240503 08:12:36.927714 5854 raft_consensus.cc:348] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.928086 5854 raft_consensus.cc:374] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.928243 5854 raft_consensus.cc:729] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.928598 5854 consensus_queue.cc:260] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:36.929379 5843 tablet.cc:1722] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:36.930374 5854 ts_tablet_manager.cc:1398] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:36.940637 5868 tablet_bootstrap.cc:492] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:36.940882 5868 ts_tablet_manager.cc:1367] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.234s user 0.006s sys 0.000s
I20240503 08:12:36.941795 5868 raft_consensus.cc:348] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:36.942171 5868 raft_consensus.cc:374] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.942366 5868 raft_consensus.cc:729] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.943403 5868 consensus_queue.cc:260] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:36.946866 5868 ts_tablet_manager.cc:1398] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:36.955890 5802 tablet_copy_client.cc:783] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:36.956338 5802 tablet_copy_client.cc:647] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:36.959010 5856 tablet_bootstrap.cc:492] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:36.959339 5856 ts_tablet_manager.cc:1367] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.238s user 0.006s sys 0.002s
I20240503 08:12:36.960706 5856 raft_consensus.cc:348] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:36.961035 5856 raft_consensus.cc:374] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:36.961155 5856 raft_consensus.cc:729] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:36.961221 5802 tablet_copy_client.cc:523] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:36.961484 5856 consensus_queue.cc:260] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:36.963603 5856 ts_tablet_manager.cc:1398] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:36.965878 5884 tablet_replica.cc:330] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:36.966183 5884 raft_consensus.cc:2229] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:36.966523 5884 raft_consensus.cc:2258] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:36.967371 5305 raft_consensus.cc:686] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:36.967844 5305 consensus_queue.cc:237] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:36.973615 5802 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:36.975203 5869 tablet_bootstrap.cc:492] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:36.976404 5884 ts_tablet_manager.cc:1875] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:36.979205 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet a752938a2b6c424db0c08bb85b64429b with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:36.983522 5865 tablet.cc:1722] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:36.994683 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=e732af17bba7429da9c7c8b3e8d4a0a2] alter complete (version 3)
I20240503 08:12:36.996273 2202 raft_consensus.cc:2932] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:36.999513 5869 ts_tablet_manager.cc:1367] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.261s user 0.002s sys 0.009s
I20240503 08:12:36.995955 5785 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:37.001698 5785 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:37.002199 5785 ts_tablet_manager.cc:1367] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.549s user 0.059s sys 0.002s
I20240503 08:12:37.003794 5785 raft_consensus.cc:348] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:37.006055 5785 raft_consensus.cc:729] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:37.006405 5785 consensus_queue.cc:260] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:37.007165 5785 ts_tablet_manager.cc:1398] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:37.008203 5737 raft_consensus.cc:482] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:37.008397 5737 raft_consensus.cc:504] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:37.009222 5737 leader_election.cc:288] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:37.009792 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caddf5e39f8440baa8ad3656ce91a8e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:37.010556 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0caddf5e39f8440baa8ad3656ce91a8e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:37.010833 1811 raft_consensus.cc:2379] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00b61625d8a7460892c118a2570afda5 in current term 1: Already voted for candidate 31f41542452b4b1eb994ff4223d039fc in this term.
I20240503 08:12:37.012181 4902 leader_election.cc:302] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e
I20240503 08:12:37.012547 5737 raft_consensus.cc:3037] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.013892 1833 tablet_copy_service.cc:342] P b8f581390da449ccb9e299011296405e: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-3219159f92774adbb7ecaa77ac267021 received from {username='slave'} at 127.27.157.197:43085
I20240503 08:12:37.014041 1833 tablet_copy_service.cc:433] P b8f581390da449ccb9e299011296405e: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-3219159f92774adbb7ecaa77ac267021 on tablet 3219159f92774adbb7ecaa77ac267021 with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:37.015354 5737 raft_consensus.cc:2726] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:37.001299 5869 raft_consensus.cc:348] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:37.016029 5869 raft_consensus.cc:374] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.018187 5869 raft_consensus.cc:729] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.027453 1670 catalog_manager.cc:5484] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:37.042033 5887 tablet_replica.cc:330] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:37.042546 5887 raft_consensus.cc:2229] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:37.042989 5887 raft_consensus.cc:2258] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:37.045167 5887 ts_tablet_manager.cc:1875] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:37.066792 5887 ts_tablet_manager.cc:1888] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:37.070506 5786 raft_consensus.cc:482] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:37.070768 5786 raft_consensus.cc:504] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:37.071657 5786 leader_election.cc:288] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:37.072245 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e641fe703c14f4cb1bd22c676d79576" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:37.072669 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e641fe703c14f4cb1bd22c676d79576" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:37.072933 1811 raft_consensus.cc:2454] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:37.073505 4769 leader_election.cc:302] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f581390da449ccb9e299011296405e, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:37.073988 5786 raft_consensus.cc:2781] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:37.075704 5786 raft_consensus.cc:482] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:37.075832 5786 raft_consensus.cc:3037] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.019178 5869 consensus_queue.cc:260] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:37.077670 4568 raft_consensus.cc:2454] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:37.079855 5869 ts_tablet_manager.cc:1398] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.080s user 0.003s sys 0.000s
I20240503 08:12:37.082479 5786 raft_consensus.cc:504] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:37.085258 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e641fe703c14f4cb1bd22c676d79576" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:37.085556 1811 raft_consensus.cc:3037] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.090903 1811 raft_consensus.cc:2454] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:37.091462 4769 leader_election.cc:302] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f581390da449ccb9e299011296405e, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:37.091816 5190 raft_consensus.cc:2781] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:37.092052 5786 leader_election.cc:288] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:37.096933 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e641fe703c14f4cb1bd22c676d79576" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:37.097223 4568 raft_consensus.cc:3037] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.100365 4568 raft_consensus.cc:2454] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:37.101155 1670 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:37.101428 1670 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:37.106926 1670 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 0e912869a85b4f708d2f3d2562920649
I20240503 08:12:37.107924 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e912869a85b4f708d2f3d2562920649 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:37 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:37.108217 4812 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e912869a85b4f708d2f3d2562920649 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:37 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:37.108999 5892 tablet_replica.cc:330] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:37.108932 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e912869a85b4f708d2f3d2562920649 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:37 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:37.109202 5892 raft_consensus.cc:2229] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:37.109488 5892 raft_consensus.cc:2258] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:37.110870 5892 ts_tablet_manager.cc:1875] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:37.116780 5869 tablet_bootstrap.cc:492] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:37.121521 5869 tablet_bootstrap.cc:654] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.122241 5893 tablet_replica.cc:330] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:37.126463 5893 raft_consensus.cc:2229] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:37.126853 5887 log.cc:1196] T a752938a2b6c424db0c08bb85b64429b P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/a752938a2b6c424db0c08bb85b64429b
I20240503 08:12:37.127175 5884 ts_tablet_manager.cc:1888] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:37.127462 5893 raft_consensus.cc:2258] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:37.128829 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet a752938a2b6c424db0c08bb85b64429b (table rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]) successfully deleted
I20240503 08:12:37.129076 5892 ts_tablet_manager.cc:1888] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:37.129211 5892 log.cc:1196] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/0e912869a85b4f708d2f3d2562920649
I20240503 08:12:37.129663 5892 ts_tablet_manager.cc:1909] T 0e912869a85b4f708d2f3d2562920649 P f24648c92c384f1ba349eee801f985e2: Deleting consensus metadata
I20240503 08:12:37.129966 5884 log.cc:1196] T 64973388b1c8466798b93ee424df489d P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/64973388b1c8466798b93ee424df489d
I20240503 08:12:37.130789 5190 raft_consensus.cc:686] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:37.130883 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet 0e912869a85b4f708d2f3d2562920649 (table rebalancer_dynamic_table_new_name [id=e732af17bba7429da9c7c8b3e8d4a0a2]) successfully deleted
I20240503 08:12:37.131457 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 64973388b1c8466798b93ee424df489d (table rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]) successfully deleted
I20240503 08:12:37.131778 5190 consensus_queue.cc:237] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:37.132277 5893 ts_tablet_manager.cc:1875] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:37.134258 5891 tablet_replica.cc:330] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:37.136515 5891 raft_consensus.cc:2229] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:37.137251 5891 raft_consensus.cc:2258] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:37.147828 5891 ts_tablet_manager.cc:1875] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:37.150835 5869 tablet_bootstrap.cc:492] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:37.151278 5869 ts_tablet_manager.cc:1367] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.003s
I20240503 08:12:37.152427 5869 raft_consensus.cc:348] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:37.152784 5869 raft_consensus.cc:374] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.152909 5869 raft_consensus.cc:729] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.153271 5869 consensus_queue.cc:260] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:37.164163 5869 ts_tablet_manager.cc:1398] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 08:12:37.168403 1672 catalog_manager.cc:5484] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:37.177455 5893 ts_tablet_manager.cc:1888] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:37.177815 5893 log.cc:1196] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/0e912869a85b4f708d2f3d2562920649
I20240503 08:12:37.178510 5893 ts_tablet_manager.cc:1909] T 0e912869a85b4f708d2f3d2562920649 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:37.179849 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 0e912869a85b4f708d2f3d2562920649 (table rebalancer_dynamic_table_new_name [id=e732af17bba7429da9c7c8b3e8d4a0a2]) successfully deleted
I20240503 08:12:37.188500 5802 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:37.190449 5891 ts_tablet_manager.cc:1888] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:37.190779 5891 log.cc:1196] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/0e912869a85b4f708d2f3d2562920649
I20240503 08:12:37.191362 5891 ts_tablet_manager.cc:1909] T 0e912869a85b4f708d2f3d2562920649 P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:37.192627 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 0e912869a85b4f708d2f3d2562920649 (table rebalancer_dynamic_table_new_name [id=e732af17bba7429da9c7c8b3e8d4a0a2]) successfully deleted
I20240503 08:12:37.192992 5802 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:12:37.193392 5802 ts_tablet_manager.cc:1367] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.221s user 0.068s sys 0.008s
I20240503 08:12:37.195024 5802 raft_consensus.cc:348] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:37.195437 5802 raft_consensus.cc:729] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:12:37.195715 5802 consensus_queue.cc:260] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:37.196368 5802 ts_tablet_manager.cc:1398] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:37.197803 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-7c37418ac97343d1931a8e1239218a38 received from {username='slave'} at 127.27.157.198:44359
I20240503 08:12:37.197988 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-7c37418ac97343d1931a8e1239218a38 on tablet 7c37418ac97343d1931a8e1239218a38 with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:37.204207 4568 raft_consensus.cc:1203] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:37.212040 5737 raft_consensus.cc:482] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:37.212224 5737 raft_consensus.cc:504] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:37.212921 5737 leader_election.cc:288] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:37.219135 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296de1897a5e4969aa36b501080a35d9" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:37.219477 2202 raft_consensus.cc:2454] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:37.220134 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296de1897a5e4969aa36b501080a35d9" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:37.220367 4568 raft_consensus.cc:2454] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:37.221017 4900 leader_election.cc:302] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:37.221334 5737 raft_consensus.cc:2781] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:37.221428 5737 raft_consensus.cc:482] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:37.221501 5737 raft_consensus.cc:3037] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.224431 5737 raft_consensus.cc:504] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:37.232403 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296de1897a5e4969aa36b501080a35d9" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:37.232856 2202 raft_consensus.cc:3037] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.235769 5737 leader_election.cc:288] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:37.236970 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296de1897a5e4969aa36b501080a35d9" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:37.237303 4568 raft_consensus.cc:3037] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.239989 1670 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:37.243534 4568 raft_consensus.cc:2454] T 296de1897a5e4969aa36b501080a35d9 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:37.244462 4900 leader_election.cc:302] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:37.249327 2202 raft_consensus.cc:2454] T 296de1897a5e4969aa36b501080a35d9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:37.250524 4702 raft_consensus.cc:1203] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:37.255904 5737 raft_consensus.cc:2781] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:37.256135 5737 raft_consensus.cc:686] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:37.256522 2054 tablet_service.cc:1467] Processing CreateTablet for tablet b999c127c42e43a998334d331ebd6313 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=263ba0e65e6a47fa91e92228a5e069e9]), partition=
I20240503 08:12:37.256444 5737 consensus_queue.cc:237] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:37.256877 4812 tablet_service.cc:1467] Processing CreateTablet for tablet b999c127c42e43a998334d331ebd6313 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=263ba0e65e6a47fa91e92228a5e069e9]), partition=
I20240503 08:12:37.256945 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b999c127c42e43a998334d331ebd6313. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.257273 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b999c127c42e43a998334d331ebd6313. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.261085 4682 tablet_service.cc:1467] Processing CreateTablet for tablet b999c127c42e43a998334d331ebd6313 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=263ba0e65e6a47fa91e92228a5e069e9]), partition=
I20240503 08:12:37.261529 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b999c127c42e43a998334d331ebd6313. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.268038 1672 catalog_manager.cc:5484] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:37.281186 5899 tablet_bootstrap.cc:492] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:37.284132 5898 tablet_bootstrap.cc:492] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:37.285434 5900 tablet_bootstrap.cc:492] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:37.287088 5899 tablet_bootstrap.cc:654] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.289323 5898 tablet_bootstrap.cc:654] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.293722 5900 tablet_bootstrap.cc:654] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.300204 5898 tablet_bootstrap.cc:492] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:37.300398 5898 ts_tablet_manager.cc:1367] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.007s
I20240503 08:12:37.302335 5898 raft_consensus.cc:348] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:37.302685 5898 raft_consensus.cc:374] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.302773 5898 raft_consensus.cc:729] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.303759 5899 tablet_bootstrap.cc:492] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:37.303953 5899 ts_tablet_manager.cc:1367] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.006s
I20240503 08:12:37.304888 5899 raft_consensus.cc:348] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:37.305217 5899 raft_consensus.cc:374] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.305346 5899 raft_consensus.cc:729] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.305636 5899 consensus_queue.cc:260] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:37.306402 5899 ts_tablet_manager.cc:1398] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:37.306710 5898 consensus_queue.cc:260] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:37.307471 5875 raft_consensus.cc:482] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:37.307638 5875 raft_consensus.cc:504] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:37.308349 5875 leader_election.cc:288] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:37.308624 5898 ts_tablet_manager.cc:1398] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 08:12:37.312119 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b999c127c42e43a998334d331ebd6313" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:37.312424 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b999c127c42e43a998334d331ebd6313" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:37.312678 4702 raft_consensus.cc:2454] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
W20240503 08:12:37.312932 2010 leader_election.cc:341] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:37.313738 2010 leader_election.cc:302] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a; no voters: f24648c92c384f1ba349eee801f985e2
I20240503 08:12:37.314129 5875 raft_consensus.cc:2781] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:37.314260 5875 raft_consensus.cc:482] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:37.314369 5875 raft_consensus.cc:3037] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.319365 5875 raft_consensus.cc:504] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:37.321767 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b999c127c42e43a998334d331ebd6313" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:37.322048 4702 raft_consensus.cc:3037] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.325181 4702 raft_consensus.cc:2454] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:37.326169 2010 leader_election.cc:302] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:37.326537 5573 raft_consensus.cc:2781] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:37.326699 5573 raft_consensus.cc:686] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:37.326979 5573 consensus_queue.cc:237] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:37.328439 5900 tablet_bootstrap.cc:492] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:37.328696 5900 ts_tablet_manager.cc:1367] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.006s
I20240503 08:12:37.329661 5900 raft_consensus.cc:348] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:37.331128 1670 catalog_manager.cc:5484] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:37.332372 5875 leader_election.cc:288] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:37.332178 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b999c127c42e43a998334d331ebd6313" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:37.334954 5900 raft_consensus.cc:374] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:37.335108 5900 raft_consensus.cc:729] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:37.335439 5900 consensus_queue.cc:260] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:37.335772 4835 raft_consensus.cc:3037] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:37.336110 5900 ts_tablet_manager.cc:1398] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:37.338871 4835 raft_consensus.cc:2454] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:37.372383 2074 consensus_queue.cc:237] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:37.374814 4967 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:12:37.375007 2009 consensus_peers.cc:485] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d4d06fdfc4e49d5b0790e2ee3b4cc83. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:37.375097 4702 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:37.375459 5573 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:37.375972 5875 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:37.378962 5573 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:37.379771 4967 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:37.384021 4702 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:37.384686 1665 catalog_manager.cc:5484] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:37.406232 4568 consensus_queue.cc:237] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:37.409051 1811 raft_consensus.cc:1261] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:37.409032 4835 raft_consensus.cc:1261] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:12:37.409127 4502 consensus_peers.cc:485] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cda734f360c4d3eac88a6517028e49d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:37.409685 5919 consensus_queue.cc:1035] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:37.410137 5650 consensus_queue.cc:1035] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:37.413125 5650 raft_consensus.cc:2932] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:37.413760 4835 raft_consensus.cc:2932] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:37.417762 1811 raft_consensus.cc:2932] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:37.419973 1670 catalog_manager.cc:5484] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:37.430577 4835 consensus_queue.cc:237] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:12:37.432621 1811 raft_consensus.cc:1261] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:37.432825 2202 raft_consensus.cc:1261] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:37.433411 5786 consensus_queue.cc:1035] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:37.433773 5890 consensus_queue.cc:1035] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:37.436441 5190 raft_consensus.cc:2932] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:37.437175 1811 raft_consensus.cc:2932] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:37.437170 2202 raft_consensus.cc:2932] T 820ecfef09a34327a08f7c3a893ccb05 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:37.441625 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 820ecfef09a34327a08f7c3a893ccb05 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:37.442837 1665 catalog_manager.cc:5484] T 820ecfef09a34327a08f7c3a893ccb05 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:37.444254 1670 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:37.448052 4947 tablet_service.cc:1514] Processing DeleteTablet for tablet 820ecfef09a34327a08f7c3a893ccb05 with delete_type TABLET_DATA_TOMBSTONED (TS 00b61625d8a7460892c118a2570afda5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:37.450567 4835 consensus_queue.cc:237] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:37.451030 5927 tablet_replica.cc:330] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:37.451256 5927 raft_consensus.cc:2229] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:37.451982 5927 raft_consensus.cc:2258] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:37.453521 5927 ts_tablet_manager.cc:1875] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:37.454066 5305 consensus_queue.cc:1035] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:37.466928 4767 consensus_peers.cc:485] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6cff57835b846e6b48c08bef52f5b68. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:37.467880 1811 raft_consensus.cc:1261] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:37.468724 5786 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:37.472338 5786 raft_consensus.cc:2932] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:37.473459 1811 raft_consensus.cc:2932] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:37.475956 5927 ts_tablet_manager.cc:1888] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:37.479951 5927 log.cc:1196] T 820ecfef09a34327a08f7c3a893ccb05 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/820ecfef09a34327a08f7c3a893ccb05
I20240503 08:12:37.480881 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 820ecfef09a34327a08f7c3a893ccb05 (table rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]) successfully deleted
I20240503 08:12:37.484297 1663 catalog_manager.cc:5484] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:37.487985 4702 raft_consensus.cc:1261] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:37.490033 4835 raft_consensus.cc:1261] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:37.490937 5573 consensus_queue.cc:1035] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:37.492033 4701 raft_consensus.cc:1261] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:37.493614 5573 consensus_queue.cc:1035] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:37.497867 5931 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 08:12:37.511960 4701 raft_consensus.cc:2932] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:37.517431 5931 consensus_queue.cc:1035] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:37.547892 5935 consensus_queue.cc:1035] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:37.553442 1670 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet b999c127c42e43a998334d331ebd6313 (table rebalancer_dynamic_table [id=263ba0e65e6a47fa91e92228a5e069e9]). Expected version 1 got 0
I20240503 08:12:37.558431 1670 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet b999c127c42e43a998334d331ebd6313 (table rebalancer_dynamic_table [id=263ba0e65e6a47fa91e92228a5e069e9]). Expected version 1 got 0
I20240503 08:12:37.559646 5786 consensus_queue.cc:1035] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:37.560211 5928 mvcc.cc:205] Tried to move back new op lower bound from 7023509329855434752 to 7023509329211973632. Current Snapshot: MvccSnapshot[applied={T|T < 7023509329855434752}]
I20240503 08:12:37.570655 5865 tablet.cc:1722] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:37.571410 5865 tablet.cc:1620] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:37.574981 5956 tablet.cc:1722] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:37.576368 5953 tablet.cc:1722] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:37.582093 2074 consensus_queue.cc:237] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:37.576097 5956 tablet.cc:1620] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:37.586782 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=263ba0e65e6a47fa91e92228a5e069e9] alter complete (version 1)
I20240503 08:12:37.587349 1663 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet b999c127c42e43a998334d331ebd6313 (table rebalancer_dynamic_table [id=263ba0e65e6a47fa91e92228a5e069e9]). Expected version 1 got 0
I20240503 08:12:37.588851 5953 tablet.cc:1620] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: MemRowSet was empty: no flush needed.
I20240503 08:12:37.605235 1670 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet b999c127c42e43a998334d331ebd6313 (table rebalancer_dynamic_table [id=263ba0e65e6a47fa91e92228a5e069e9]). Expected version 1 got 0
I20240503 08:12:37.616883 1943 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:37.618300 2202 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:37.620669 5357 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:37.621603 5357 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:12:37.625015 2010 consensus_peers.cc:485] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a92cb30e806b4e58a2d9f9cfa516bdf9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:37.639672 5357 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:37.640832 2201 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:37.645233 1943 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:37.645728 1663 catalog_manager.cc:5484] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:37.676795 1943 consensus_queue.cc:237] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:37.685091 1812 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:37.685487 1878 consensus_peers.cc:485] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 733bed28775e4779b74e92d8a395234f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:37.685900 2201 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:37.685899 5968 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:37.686455 5863 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:37.689874 5417 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:37.691903 2201 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:37.697619 1668 catalog_manager.cc:5484] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:37.711133 1812 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:37.716272 1663 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:37.723461 5814 raft_consensus.cc:1050] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:37.724385 5814 consensus_queue.cc:237] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:37.730486 4568 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:37.731235 5879 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:37.731649 2074 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:37.732982 1943 consensus_queue.cc:237] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:37.733139 5953 tablet.cc:1722] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:37.733637 5953 tablet.cc:1620] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: MemRowSet was empty: no flush needed.
I20240503 08:12:37.733803 5977 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:37.734529 1944 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:37.735627 5977 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:37.736707 5896 consensus_queue.cc:1035] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:37.744056 1877 consensus_peers.cc:485] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4900ffdbc624376b32f054efdb100b5. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:37.744138 4967 raft_consensus.cc:1261] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:37.745697 5968 consensus_queue.cc:1035] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:37.748019 5865 tablet.cc:1722] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:37.748706 5896 consensus_queue.cc:1035] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:37.751173 5865 tablet.cc:1620] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:37.747260 2074 raft_consensus.cc:1261] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:37.753888 1663 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=263ba0e65e6a47fa91e92228a5e069e9] alter complete (version 2)
I20240503 08:12:37.754184 5968 consensus_queue.cc:1035] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:37.761143 5956 tablet.cc:1722] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:37.761734 5956 tablet.cc:1620] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:37.778553 5879 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:37.782148 4569 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:37.800364 1943 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:37.804046 5417 raft_consensus.cc:2932] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:37.805166 1663 catalog_manager.cc:5484] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:37.810247 2073 raft_consensus.cc:2932] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:37.812275 4967 raft_consensus.cc:2932] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:37.813004 1668 catalog_manager.cc:5484] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:37.833200 5989 ts_tablet_manager.cc:902] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a: Initiating tablet copy from peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:37.837308 5968 raft_consensus.cc:1050] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:12:37.839259 5968 consensus_queue.cc:237] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:37.842532 4701 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:37.842847 2201 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:37.843420 5417 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:37.844058 5968 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:37.844501 2074 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:37.846613 5968 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:37.855733 2073 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:37.857797 1668 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:37.870441 5953 tablet.cc:1722] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:37.875581 5865 tablet.cc:1722] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:37.878865 5417 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:37.883396 1671 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:37.884404 4702 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:37.888151 5956 tablet.cc:1722] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:37.894320 1664 catalog_manager.cc:5484] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:37.900043 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=263ba0e65e6a47fa91e92228a5e069e9] alter complete (version 3)
I20240503 08:12:37.905100 2202 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:37.915141 6002 ts_tablet_manager.cc:902] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:37.925174 1812 consensus_queue.cc:237] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:37.930162 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 9c02a6bb7f7d4b42a6af774f342b0398 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.930660 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c02a6bb7f7d4b42a6af774f342b0398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.931931 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 9c02a6bb7f7d4b42a6af774f342b0398 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.932492 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c02a6bb7f7d4b42a6af774f342b0398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.934072 4947 tablet_service.cc:1467] Processing CreateTablet for tablet 09a268c672ba4580a2d93b5d15d12666 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.934085 4683 tablet_service.cc:1467] Processing CreateTablet for tablet df0c416025c74075b6429d169fcdd58c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.934522 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0c416025c74075b6429d169fcdd58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.934520 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a268c672ba4580a2d93b5d15d12666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.937305 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 9c02a6bb7f7d4b42a6af774f342b0398 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.937896 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c02a6bb7f7d4b42a6af774f342b0398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.941358 4548 tablet_service.cc:1467] Processing CreateTablet for tablet df0c416025c74075b6429d169fcdd58c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.941852 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0c416025c74075b6429d169fcdd58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.941987 1943 consensus_queue.cc:237] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:37.942773 4814 tablet_service.cc:1467] Processing CreateTablet for tablet df0c416025c74075b6429d169fcdd58c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.943274 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0c416025c74075b6429d169fcdd58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.943480 4547 tablet_service.cc:1467] Processing CreateTablet for tablet 09a268c672ba4580a2d93b5d15d12666 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.943908 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a268c672ba4580a2d93b5d15d12666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.943943 4813 tablet_service.cc:1467] Processing CreateTablet for tablet 09a268c672ba4580a2d93b5d15d12666 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:37.947583 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a268c672ba4580a2d93b5d15d12666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:37.948838 2202 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:37.949386 4702 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:37.949565 5863 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:37.950096 5417 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:37.952941 5417 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:37.953281 4567 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:37.955065 5977 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:37.956404 2202 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:37.956527 4702 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:37.962319 5977 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:37.962999 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c37418ac97343d1931a8e1239218a38 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:37.966857 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3219159f92774adbb7ecaa77ac267021 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:37.969985 1665 catalog_manager.cc:5484] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:37.975152 6011 tablet_bootstrap.cc:492] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:37.976487 1664 catalog_manager.cc:5484] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:37.987291 6011 tablet_bootstrap.cc:654] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.987636 6007 tablet_bootstrap.cc:492] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:37.990197 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 3219159f92774adbb7ecaa77ac267021 with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:37.990288 6008 tablet_bootstrap.cc:492] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:37.992820 6008 tablet_bootstrap.cc:654] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.993541 6012 tablet_bootstrap.cc:492] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:37.995838 6012 tablet_bootstrap.cc:654] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:37.998261 4569 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:38.000044 6007 tablet_bootstrap.cc:654] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.001621 6011 tablet_bootstrap.cc:492] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:38.004390 6011 ts_tablet_manager.cc:1367] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.005s
I20240503 08:12:38.005926 6011 raft_consensus.cc:348] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.008081 6011 raft_consensus.cc:374] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.009570 6011 raft_consensus.cc:729] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.010025 6011 consensus_queue.cc:260] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.010926 6011 ts_tablet_manager.cc:1398] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:38.011379 6008 tablet_bootstrap.cc:492] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:38.012883 6007 tablet_bootstrap.cc:492] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:38.014461 6008 ts_tablet_manager.cc:1367] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240503 08:12:38.016475 6008 raft_consensus.cc:348] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.019269 6012 tablet_bootstrap.cc:492] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:38.019832 6007 ts_tablet_manager.cc:1367] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240503 08:12:38.020205 6012 ts_tablet_manager.cc:1367] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.004s
I20240503 08:12:38.025100 6007 raft_consensus.cc:348] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.026619 6012 raft_consensus.cc:348] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.027222 6008 raft_consensus.cc:374] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.030969 1667 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:38.031352 6012 raft_consensus.cc:374] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.031304 1667 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:38.032692 6008 raft_consensus.cc:729] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.032980 6007 raft_consensus.cc:374] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.033502 6007 raft_consensus.cc:729] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.033715 6012 raft_consensus.cc:729] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.034132 6008 consensus_queue.cc:260] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.036424 6007 consensus_queue.cc:260] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.040284 5988 ts_tablet_manager.cc:902] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:38.040567 1667 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet b999c127c42e43a998334d331ebd6313
I20240503 08:12:38.043246 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet b999c127c42e43a998334d331ebd6313 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:38 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:38.057510 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet b999c127c42e43a998334d331ebd6313 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:38 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:38.058722 6008 ts_tablet_manager.cc:1398] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.043s user 0.001s sys 0.002s
I20240503 08:12:38.036882 6012 consensus_queue.cc:260] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.062243 6012 ts_tablet_manager.cc:1398] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.040s user 0.001s sys 0.002s
I20240503 08:12:38.062721 6007 ts_tablet_manager.cc:1398] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.042s user 0.000s sys 0.003s
I20240503 08:12:38.063701 6007 tablet_bootstrap.cc:492] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:38.067018 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c37418ac97343d1931a8e1239218a38 with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:38.067587 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet b999c127c42e43a998334d331ebd6313 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:38 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:38.063290 6012 tablet_bootstrap.cc:492] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:38.067838 2072 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:38.072335 5879 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:38.073405 5989 tablet_copy_client.cc:308] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.197:42465
I20240503 08:12:38.074797 2072 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:38.078287 6012 tablet_bootstrap.cc:654] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.079232 4589 tablet_copy_service.cc:140] P 5f95b05d111e4090bbc59dec394ed0ab: Received BeginTabletCopySession request for tablet 3cda734f360c4d3eac88a6517028e49d from peer 43e2e01cdd314f9c872c4336ef74250a ({username='slave'} at 127.27.157.195:44181)
I20240503 08:12:38.079517 4589 tablet_copy_service.cc:161] P 5f95b05d111e4090bbc59dec394ed0ab: Beginning new tablet copy session on tablet 3cda734f360c4d3eac88a6517028e49d from peer 43e2e01cdd314f9c872c4336ef74250a at {username='slave'} at 127.27.157.195:44181: session id = 43e2e01cdd314f9c872c4336ef74250a-3cda734f360c4d3eac88a6517028e49d
I20240503 08:12:38.083665 2073 consensus_queue.cc:237] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:38.096307 4589 tablet_copy_source_session.cc:205] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:38.098584 6013 tablet_replica.cc:330] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:38.098942 6013 raft_consensus.cc:2229] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:38.099241 6013 raft_consensus.cc:2258] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:38.099927 6010 tablet_bootstrap.cc:492] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:38.102329 6010 tablet_bootstrap.cc:654] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.103120 5919 raft_consensus.cc:482] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:38.103394 5919 raft_consensus.cc:504] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.103904 1942 raft_consensus.cc:1261] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:38.104151 5919 leader_election.cc:288] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:38.104565 5875 consensus_queue.cc:1035] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:38.104813 2202 raft_consensus.cc:1261] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:38.105417 6013 ts_tablet_manager.cc:1875] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:38.105894 5875 consensus_queue.cc:1035] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:38.079301 6007 tablet_bootstrap.cc:654] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.106896 5989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cda734f360c4d3eac88a6517028e49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.095753 6023 ts_tablet_manager.cc:902] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:38.108106 6022 tablet_replica.cc:330] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:38.102833 6020 tablet_replica.cc:330] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
W20240503 08:12:38.108536 2010 consensus_peers.cc:485] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 266d0c75717b4fc98eae8b75a5ba4d8c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:38.109288 6022 raft_consensus.cc:2229] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:38.109433 6020 raft_consensus.cc:2229] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:38.109809 6020 raft_consensus.cc:2258] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:38.110502 6022 pending_rounds.cc:62] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a: Trying to abort 1 pending ops.
I20240503 08:12:38.111012 6020 ts_tablet_manager.cc:1875] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:38.106308 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c416025c74075b6429d169fcdd58c" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
W20240503 08:12:38.112046 4504 leader_election.cc:341] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:38.111263 6022 pending_rounds.cc:68] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023509331305885696 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7c37418ac97343d1931a8e1239218a38" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } } }
I20240503 08:12:38.113829 6022 raft_consensus.cc:2894] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 08:12:38.114442 6022 raft_consensus.cc:2866] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:12:38.114907 6022 raft_consensus.cc:2258] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:38.115454 4833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c416025c74075b6429d169fcdd58c" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
W20240503 08:12:38.116528 4504 leader_election.cc:341] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:38.116784 4504 leader_election.cc:302] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab; no voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2
I20240503 08:12:38.117206 5919 raft_consensus.cc:2726] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:38.119875 6011 tablet_bootstrap.cc:492] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:38.121001 6021 tablet_replica.cc:330] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:38.121053 6022 ts_tablet_manager.cc:1875] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:38.121351 6021 raft_consensus.cc:2229] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:38.121798 6021 raft_consensus.cc:2258] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:38.123499 6021 ts_tablet_manager.cc:1875] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:38.128284 6011 tablet_bootstrap.cc:654] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.150295 1664 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:38.163878 4548 tablet_service.cc:1467] Processing CreateTablet for tablet ad7cc4af7e954cb89b16ab7f518d24f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4906818a88ff460cbb11ce4f629fe500]), partition=
I20240503 08:12:38.164157 4814 tablet_service.cc:1467] Processing CreateTablet for tablet ad7cc4af7e954cb89b16ab7f518d24f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4906818a88ff460cbb11ce4f629fe500]), partition=
I20240503 08:12:38.164630 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7cc4af7e954cb89b16ab7f518d24f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.164719 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7cc4af7e954cb89b16ab7f518d24f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.165854 4947 tablet_service.cc:1467] Processing CreateTablet for tablet ad7cc4af7e954cb89b16ab7f518d24f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4906818a88ff460cbb11ce4f629fe500]), partition=
I20240503 08:12:38.166019 6012 tablet_bootstrap.cc:492] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:38.166460 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7cc4af7e954cb89b16ab7f518d24f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.175633 6007 tablet_bootstrap.cc:492] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:38.176057 6007 ts_tablet_manager.cc:1367] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.112s user 0.007s sys 0.001s
I20240503 08:12:38.179716 6012 ts_tablet_manager.cc:1367] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.116s user 0.000s sys 0.008s
I20240503 08:12:38.181126 6012 raft_consensus.cc:348] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.182039 6033 ts_tablet_manager.cc:902] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:38.177484 6007 raft_consensus.cc:348] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.183109 6007 raft_consensus.cc:374] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.183269 6007 raft_consensus.cc:729] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.183696 6007 consensus_queue.cc:260] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.184397 6007 ts_tablet_manager.cc:1398] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:12:38.184823 6007 tablet_bootstrap.cc:492] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:38.184521 5875 raft_consensus.cc:2932] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:38.185240 6010 tablet_bootstrap.cc:492] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:38.185606 6010 ts_tablet_manager.cc:1367] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.086s user 0.000s sys 0.005s
I20240503 08:12:38.186467 6007 tablet_bootstrap.cc:654] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.190443 6011 tablet_bootstrap.cc:492] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:38.190809 2201 raft_consensus.cc:2932] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:38.191453 6012 raft_consensus.cc:374] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.191684 6012 raft_consensus.cc:729] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.192224 1943 raft_consensus.cc:2932] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:38.192078 6012 consensus_queue.cc:260] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.195637 6012 ts_tablet_manager.cc:1398] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.015s user 0.001s sys 0.002s
I20240503 08:12:38.195567 6010 raft_consensus.cc:348] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.195794 6010 raft_consensus.cc:374] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.195899 6010 raft_consensus.cc:729] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.198882 6022 ts_tablet_manager.cc:1888] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:38.199518 6011 ts_tablet_manager.cc:1367] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.002s
I20240503 08:12:38.200009 1668 catalog_manager.cc:5484] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:38.201377 6011 raft_consensus.cc:348] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.203936 6008 tablet_bootstrap.cc:492] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:38.205605 6008 tablet_bootstrap.cc:654] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.210150 6002 tablet_copy_client.cc:308] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:12:38.210068 6010 consensus_queue.cc:260] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.212424 6020 ts_tablet_manager.cc:1888] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:38.211093 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet f6cff57835b846e6b48c08bef52f5b68 from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:60523)
I20240503 08:12:38.215413 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet f6cff57835b846e6b48c08bef52f5b68 from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:60523: session id = 5f95b05d111e4090bbc59dec394ed0ab-f6cff57835b846e6b48c08bef52f5b68
I20240503 08:12:38.219609 6020 log.cc:1196] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/b999c127c42e43a998334d331ebd6313
I20240503 08:12:38.219261 4855 tablet_copy_source_session.cc:205] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:38.220535 5948 raft_consensus.cc:482] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:38.220719 5948 raft_consensus.cc:504] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.221632 5948 leader_election.cc:288] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:38.222283 6011 raft_consensus.cc:374] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.222409 6011 raft_consensus.cc:729] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.222728 6011 consensus_queue.cc:260] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.223212 4833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c02a6bb7f7d4b42a6af774f342b0398" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:38.223385 6011 ts_tablet_manager.cc:1398] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20240503 08:12:38.223531 4833 raft_consensus.cc:2454] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:38.223907 6011 tablet_bootstrap.cc:492] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:38.224107 4638 leader_election.cc:302] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:38.224534 5948 raft_consensus.cc:2781] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:38.224656 5948 raft_consensus.cc:482] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:38.224735 5948 raft_consensus.cc:3037] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.226589 6011 tablet_bootstrap.cc:654] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.227948 6010 ts_tablet_manager.cc:1398] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240503 08:12:38.222088 5988 tablet_copy_client.cc:308] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:38.228497 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c02a6bb7f7d4b42a6af774f342b0398" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:38.229250 5948 raft_consensus.cc:504] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.230794 5948 leader_election.cc:288] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:38.231185 4833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c02a6bb7f7d4b42a6af774f342b0398" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:38.231647 4833 raft_consensus.cc:3037] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.232194 1942 raft_consensus.cc:2454] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:38.233976 6002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cff57835b846e6b48c08bef52f5b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.234599 4833 raft_consensus.cc:2454] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:38.235024 4638 leader_election.cc:302] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:38.237394 5948 raft_consensus.cc:2781] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:38.238039 5948 raft_consensus.cc:686] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:38.238593 5948 consensus_queue.cc:237] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.246331 6021 ts_tablet_manager.cc:1888] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:38.247049 6023 tablet_copy_client.cc:308] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:38.248634 6021 log.cc:1196] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/b999c127c42e43a998334d331ebd6313
I20240503 08:12:38.249753 6021 ts_tablet_manager.cc:1909] T b999c127c42e43a998334d331ebd6313 P f24648c92c384f1ba349eee801f985e2: Deleting consensus metadata
I20240503 08:12:38.251102 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet b999c127c42e43a998334d331ebd6313 (table rebalancer_dynamic_table_new_name [id=263ba0e65e6a47fa91e92228a5e069e9]) successfully deleted
I20240503 08:12:38.231120 6013 ts_tablet_manager.cc:1888] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:38.231161 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c02a6bb7f7d4b42a6af774f342b0398" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:38.251763 6013 log.cc:1196] T 3219159f92774adbb7ecaa77ac267021 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/3219159f92774adbb7ecaa77ac267021
I20240503 08:12:38.251876 1943 raft_consensus.cc:3037] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.253319 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 3219159f92774adbb7ecaa77ac267021 (table rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]) successfully deleted
I20240503 08:12:38.255017 1943 raft_consensus.cc:2454] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:38.255827 6008 tablet_bootstrap.cc:492] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:38.256109 6008 ts_tablet_manager.cc:1367] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.004s
I20240503 08:12:38.257436 6020 ts_tablet_manager.cc:1909] T b999c127c42e43a998334d331ebd6313 P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:38.258426 6047 ts_tablet_manager.cc:902] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:38.258808 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet b999c127c42e43a998334d331ebd6313 (table rebalancer_dynamic_table_new_name [id=263ba0e65e6a47fa91e92228a5e069e9]) successfully deleted
I20240503 08:12:38.257251 6008 raft_consensus.cc:348] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.259811 6008 raft_consensus.cc:374] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.259904 6008 raft_consensus.cc:729] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.205111 5989 tablet_copy_client.cc:783] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:38.229182 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a ({username='slave'} at 127.27.157.194:46923)
I20240503 08:12:38.263690 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a at {username='slave'} at 127.27.157.194:46923: session id = 315f3f0bcbf447a4b93b1aafcfbf0f5a-2d4d06fdfc4e49d5b0790e2ee3b4cc83
I20240503 08:12:38.265760 1667 catalog_manager.cc:5484] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:38.266800 6011 tablet_bootstrap.cc:492] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:38.266872 2093 tablet_copy_source_session.cc:205] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:38.266963 6011 ts_tablet_manager.cc:1367] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20240503 08:12:38.247777 2094 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet a92cb30e806b4e58a2d9f9cfa516bdf9 from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:35383)
I20240503 08:12:38.267580 2094 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet a92cb30e806b4e58a2d9f9cfa516bdf9 from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:35383: session id = 31f41542452b4b1eb994ff4223d039fc-a92cb30e806b4e58a2d9f9cfa516bdf9
I20240503 08:12:38.267867 6011 raft_consensus.cc:348] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.268095 6011 raft_consensus.cc:374] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.268183 6011 raft_consensus.cc:729] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.268448 6011 consensus_queue.cc:260] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.269197 6011 ts_tablet_manager.cc:1398] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:38.270427 6011 tablet_bootstrap.cc:492] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:38.270649 2094 tablet_copy_source_session.cc:205] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:38.270880 6008 consensus_queue.cc:260] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.272776 5989 tablet_copy_client.cc:647] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:38.273502 6023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92cb30e806b4e58a2d9f9cfa516bdf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.276319 6011 tablet_bootstrap.cc:654] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.279706 6022 log.cc:1196] T 7c37418ac97343d1931a8e1239218a38 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/7c37418ac97343d1931a8e1239218a38
I20240503 08:12:38.280104 5989 tablet_copy_client.cc:523] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:38.280815 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 7c37418ac97343d1931a8e1239218a38 (table rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]) successfully deleted
I20240503 08:12:38.281334 6022 tablet_replica.cc:330] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:38.281625 6022 raft_consensus.cc:2229] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:38.282076 6022 raft_consensus.cc:2258] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:38.283123 6022 ts_tablet_manager.cc:1875] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:38.285229 6007 tablet_bootstrap.cc:492] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:38.285454 6007 ts_tablet_manager.cc:1367] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.000s
I20240503 08:12:38.286237 6007 raft_consensus.cc:348] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.286511 6007 raft_consensus.cc:374] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.287160 6007 raft_consensus.cc:729] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.288368 6007 consensus_queue.cc:260] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.289516 6008 ts_tablet_manager.cc:1398] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.033s user 0.003s sys 0.000s
I20240503 08:12:38.292874 5988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.293617 6002 tablet_copy_client.cc:783] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:38.293900 6002 tablet_copy_client.cc:647] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:38.296576 6002 tablet_copy_client.cc:523] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:38.300550 6002 tablet_bootstrap.cc:492] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:38.301847 5988 tablet_copy_client.cc:783] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:38.302243 5988 tablet_copy_client.cc:647] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:38.302726 6033 tablet_copy_client.cc:308] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:38.303577 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 733bed28775e4779b74e92d8a395234f from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:43591)
I20240503 08:12:38.303747 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 733bed28775e4779b74e92d8a395234f from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:43591: session id = 00b61625d8a7460892c118a2570afda5-733bed28775e4779b74e92d8a395234f
I20240503 08:12:38.304538 5988 tablet_copy_client.cc:523] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:38.307559 1964 tablet_copy_source_session.cc:205] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:38.309159 6033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733bed28775e4779b74e92d8a395234f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.313714 5988 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:38.316170 6023 tablet_copy_client.cc:783] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:38.316581 6023 tablet_copy_client.cc:647] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:38.324147 5896 raft_consensus.cc:482] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:38.324342 5896 raft_consensus.cc:504] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.325136 5896 leader_election.cc:288] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:38.325913 4833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a268c672ba4580a2d93b5d15d12666" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:38.326151 4833 raft_consensus.cc:2454] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:38.326743 4903 leader_election.cc:302] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:38.291770 5989 tablet_bootstrap.cc:492] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:38.327553 5896 raft_consensus.cc:2781] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:38.327679 5896 raft_consensus.cc:482] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:38.327770 5896 raft_consensus.cc:3037] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.330619 5896 raft_consensus.cc:504] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.332211 4833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a268c672ba4580a2d93b5d15d12666" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:38.332453 4833 raft_consensus.cc:3037] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.335857 6023 tablet_copy_client.cc:523] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:38.336313 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a268c672ba4580a2d93b5d15d12666" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:38.336558 4567 raft_consensus.cc:3037] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.338709 5786 raft_consensus.cc:482] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:38.338914 5786 raft_consensus.cc:504] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.339670 5786 leader_election.cc:288] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:38.339830 4567 raft_consensus.cc:2454] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:38.340394 4900 leader_election.cc:302] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:38.340732 6007 ts_tablet_manager.cc:1398] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20240503 08:12:38.340771 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c416025c74075b6429d169fcdd58c" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:38.341022 4568 raft_consensus.cc:2454] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:38.341687 5737 raft_consensus.cc:2781] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:38.341859 5737 raft_consensus.cc:686] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:38.342165 5737 consensus_queue.cc:237] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.344655 4767 leader_election.cc:302] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:38.347966 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c416025c74075b6429d169fcdd58c" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:38.348255 4702 raft_consensus.cc:2454] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:38.348459 6023 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:38.349144 5786 raft_consensus.cc:2781] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:38.349269 5786 raft_consensus.cc:482] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:38.349371 5786 raft_consensus.cc:3037] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.349664 5988 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:38.350049 5988 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap complete.
I20240503 08:12:38.350256 5988 ts_tablet_manager.cc:1367] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.003s
I20240503 08:12:38.351558 6047 tablet_copy_client.cc:308] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:38.351819 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a268c672ba4580a2d93b5d15d12666" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:38.351501 5988 raft_consensus.cc:348] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:38.352006 5988 raft_consensus.cc:729] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: LEARNER
I20240503 08:12:38.352113 4569 raft_consensus.cc:2362] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1. Re-sending same reply.
I20240503 08:12:38.352494 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet f4900ffdbc624376b32f054efdb100b5 from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:32893)
I20240503 08:12:38.352414 5988 consensus_queue.cc:260] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:38.352670 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet f4900ffdbc624376b32f054efdb100b5 from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:32893: session id = 5f95b05d111e4090bbc59dec394ed0ab-f4900ffdbc624376b32f054efdb100b5
I20240503 08:12:38.353960 5786 raft_consensus.cc:504] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.354244 5989 tablet_bootstrap.cc:492] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:38.356065 5989 tablet_bootstrap.cc:492] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap complete.
I20240503 08:12:38.356401 5989 ts_tablet_manager.cc:1367] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.002s
I20240503 08:12:38.357555 5989 raft_consensus.cc:348] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:38.357882 5989 raft_consensus.cc:729] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: LEARNER
I20240503 08:12:38.358126 5989 consensus_queue.cc:260] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:38.359082 5989 ts_tablet_manager.cc:1398] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:38.357270 1964 tablet_copy_source_session.cc:205] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:38.359947 4589 tablet_copy_service.cc:342] P 5f95b05d111e4090bbc59dec394ed0ab: Request end of tablet copy session 43e2e01cdd314f9c872c4336ef74250a-3cda734f360c4d3eac88a6517028e49d received from {username='slave'} at 127.27.157.195:44181
I20240503 08:12:38.360174 4589 tablet_copy_service.cc:433] P 5f95b05d111e4090bbc59dec394ed0ab: ending tablet copy session 43e2e01cdd314f9c872c4336ef74250a-3cda734f360c4d3eac88a6517028e49d on tablet 3cda734f360c4d3eac88a6517028e49d with peer 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:12:38.361608 6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4900ffdbc624376b32f054efdb100b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.363798 4833 raft_consensus.cc:2454] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:38.364730 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c416025c74075b6429d169fcdd58c" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:38.364959 4569 raft_consensus.cc:3037] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.365626 5988 ts_tablet_manager.cc:1398] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 08:12:38.367384 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-2d4d06fdfc4e49d5b0790e2ee3b4cc83 received from {username='slave'} at 127.27.157.194:46923
I20240503 08:12:38.367539 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-2d4d06fdfc4e49d5b0790e2ee3b4cc83 on tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 with peer 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:12:38.374533 4569 raft_consensus.cc:2454] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:38.374969 4767 leader_election.cc:302] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:38.375500 6019 raft_consensus.cc:2781] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:38.375641 6019 raft_consensus.cc:686] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:38.375908 6019 consensus_queue.cc:237] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:38.379848 6022 ts_tablet_manager.cc:1888] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:38.380088 6022 log.cc:1196] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/b999c127c42e43a998334d331ebd6313
I20240503 08:12:38.380714 6022 ts_tablet_manager.cc:1909] T b999c127c42e43a998334d331ebd6313 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:38.380571 1667 catalog_manager.cc:5484] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:38.382062 6011 tablet_bootstrap.cc:492] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:38.382369 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df0c416025c74075b6429d169fcdd58c" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:38.382522 6033 tablet_copy_client.cc:783] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:38.382628 4702 raft_consensus.cc:3037] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.385614 4702 raft_consensus.cc:2454] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:38.386694 5896 leader_election.cc:288] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:38.387573 6033 tablet_copy_client.cc:647] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:38.390432 6011 ts_tablet_manager.cc:1367] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.120s user 0.015s sys 0.001s
I20240503 08:12:38.387125 5786 leader_election.cc:288] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:38.391752 6033 tablet_copy_client.cc:523] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:38.394435 6047 tablet_copy_client.cc:783] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:38.397725 1668 catalog_manager.cc:5484] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:38.398452 6033 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:38.402868 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet b999c127c42e43a998334d331ebd6313 (table rebalancer_dynamic_table_new_name [id=263ba0e65e6a47fa91e92228a5e069e9]) successfully deleted
I20240503 08:12:38.391453 6011 raft_consensus.cc:348] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.403165 6011 raft_consensus.cc:374] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.405462 6047 tablet_copy_client.cc:647] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:38.407565 6011 raft_consensus.cc:729] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.408499 6047 tablet_copy_client.cc:523] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:38.408671 6011 consensus_queue.cc:260] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.413964 6011 ts_tablet_manager.cc:1398] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240503 08:12:38.414499 6047 tablet_bootstrap.cc:492] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:38.423396 6002 tablet_bootstrap.cc:492] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:38.431922 6002 tablet_bootstrap.cc:492] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:38.432233 6002 ts_tablet_manager.cc:1367] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.132s user 0.005s sys 0.004s
I20240503 08:12:38.433542 6002 raft_consensus.cc:348] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:38.433826 6002 raft_consensus.cc:729] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:38.434098 6002 consensus_queue.cc:260] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:38.434679 6002 ts_tablet_manager.cc:1398] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:38.435568 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-f6cff57835b846e6b48c08bef52f5b68 received from {username='slave'} at 127.27.157.197:60523
I20240503 08:12:38.435714 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-f6cff57835b846e6b48c08bef52f5b68 on tablet f6cff57835b846e6b48c08bef52f5b68 with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:38.458460 6047 tablet_bootstrap.cc:492] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:38.459483 6047 tablet_bootstrap.cc:492] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:38.459869 6047 ts_tablet_manager.cc:1367] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.000s
I20240503 08:12:38.460871 6047 raft_consensus.cc:348] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:38.461325 6047 raft_consensus.cc:729] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:38.461642 6047 consensus_queue.cc:260] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:38.467185 6047 ts_tablet_manager.cc:1398] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:38.476321 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-f4900ffdbc624376b32f054efdb100b5 received from {username='slave'} at 127.27.157.197:32893
I20240503 08:12:38.476508 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-f4900ffdbc624376b32f054efdb100b5 on tablet f4900ffdbc624376b32f054efdb100b5 with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:38.515650 6078 ts_tablet_manager.cc:902] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:38.517711 6023 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:38.518361 6023 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:12:38.518783 6023 ts_tablet_manager.cc:1367] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.170s user 0.056s sys 0.006s
I20240503 08:12:38.520176 6023 raft_consensus.cc:348] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:38.520582 6023 raft_consensus.cc:729] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:12:38.520967 6023 consensus_queue.cc:260] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:38.523183 6023 ts_tablet_manager.cc:1398] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 08:12:38.524127 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-a92cb30e806b4e58a2d9f9cfa516bdf9 received from {username='slave'} at 127.27.157.198:35383
I20240503 08:12:38.524325 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-a92cb30e806b4e58a2d9f9cfa516bdf9 on tablet a92cb30e806b4e58a2d9f9cfa516bdf9 with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:38.530905 6033 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:38.533100 6033 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:12:38.533450 6033 ts_tablet_manager.cc:1367] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.135s user 0.045s sys 0.009s
I20240503 08:12:38.534343 6033 raft_consensus.cc:348] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:38.534628 6033 raft_consensus.cc:729] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:12:38.534885 6033 consensus_queue.cc:260] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:38.535547 6033 ts_tablet_manager.cc:1398] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:38.538022 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-733bed28775e4779b74e92d8a395234f received from {username='slave'} at 127.27.157.200:43591
I20240503 08:12:38.538187 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 00b61625d8a7460892c118a2570afda5-733bed28775e4779b74e92d8a395234f on tablet 733bed28775e4779b74e92d8a395234f with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:12:38.543637 6078 tablet_copy_client.cc:308] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:38.544421 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 266d0c75717b4fc98eae8b75a5ba4d8c from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:52311)
I20240503 08:12:38.544646 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 266d0c75717b4fc98eae8b75a5ba4d8c from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:52311: session id = f24648c92c384f1ba349eee801f985e2-266d0c75717b4fc98eae8b75a5ba4d8c
I20240503 08:12:38.547456 2093 tablet_copy_source_session.cc:205] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:38.547924 5919 raft_consensus.cc:482] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:38.548125 5919 raft_consensus.cc:504] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.548847 5919 leader_election.cc:288] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:38.548954 6078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266d0c75717b4fc98eae8b75a5ba4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.549541 4833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7cc4af7e954cb89b16ab7f518d24f8" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:38.549815 4833 raft_consensus.cc:2454] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:38.550202 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7cc4af7e954cb89b16ab7f518d24f8" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:38.550419 4967 raft_consensus.cc:2454] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:38.550792 4504 leader_election.cc:302] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:38.551694 5919 raft_consensus.cc:2781] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:38.551795 5919 raft_consensus.cc:482] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:38.551885 5919 raft_consensus.cc:3037] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.554414 6078 tablet_copy_client.cc:783] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:38.554689 6078 tablet_copy_client.cc:647] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:38.555445 5919 raft_consensus.cc:504] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.556893 6078 tablet_copy_client.cc:523] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:38.556939 4833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7cc4af7e954cb89b16ab7f518d24f8" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:38.557201 4833 raft_consensus.cc:3037] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.560421 6078 tablet_bootstrap.cc:492] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:38.564249 4833 raft_consensus.cc:2454] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:38.566053 4504 leader_election.cc:302] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:38.566524 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7cc4af7e954cb89b16ab7f518d24f8" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:38.566759 4967 raft_consensus.cc:3037] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:38.569376 4967 raft_consensus.cc:2454] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:38.571688 5919 leader_election.cc:288] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:38.571925 5919 raft_consensus.cc:2781] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:38.572463 5919 raft_consensus.cc:686] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:38.572798 5919 consensus_queue.cc:237] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:38.584728 1667 catalog_manager.cc:5484] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:38.599120 2073 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1811bd652d2c4afb9f52bcd64ceb2acb"
dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51060
I20240503 08:12:38.599448 2073 raft_consensus.cc:595] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:38.609306 5875 raft_consensus.cc:979] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: : Instructing follower 184c543cb05d49cc98ba1c56617be7a5 to start an election
I20240503 08:12:38.609411 2073 raft_consensus.cc:1203] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:38.609531 5875 raft_consensus.cc:1067] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Signalling peer 184c543cb05d49cc98ba1c56617be7a5 to start an election
I20240503 08:12:38.612232 2202 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1811bd652d2c4afb9f52bcd64ceb2acb"
dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
from {username='slave'} at 127.27.157.195:41175
I20240503 08:12:38.612558 2202 raft_consensus.cc:482] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:38.612744 2202 raft_consensus.cc:3037] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:38.621429 2202 raft_consensus.cc:504] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:38.623411 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811bd652d2c4afb9f52bcd64ceb2acb" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:38.623735 1812 raft_consensus.cc:3037] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:38.629230 1812 raft_consensus.cc:2454] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 2.
I20240503 08:12:38.633706 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811bd652d2c4afb9f52bcd64ceb2acb" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:38.634007 2073 raft_consensus.cc:3032] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:38.634117 2073 raft_consensus.cc:729] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:38.634369 4569 raft_consensus.cc:1203] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:38.634518 2073 consensus_queue.cc:260] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:38.635099 2073 raft_consensus.cc:3037] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:38.636201 6078 tablet_bootstrap.cc:492] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:38.636240 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1811bd652d2c4afb9f52bcd64ceb2acb" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:38.636476 4702 raft_consensus.cc:3037] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:38.637246 6078 tablet_bootstrap.cc:492] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:12:38.637568 6078 ts_tablet_manager.cc:1367] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.077s user 0.047s sys 0.025s
I20240503 08:12:38.638383 6078 raft_consensus.cc:348] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:38.638644 6078 raft_consensus.cc:729] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:12:38.638831 6078 consensus_queue.cc:260] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:38.638957 2073 raft_consensus.cc:2454] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 2.
I20240503 08:12:38.639633 2138 leader_election.cc:302] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:38.640017 6009 raft_consensus.cc:2781] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:38.640269 2202 leader_election.cc:288] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 2 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:38.647926 6009 raft_consensus.cc:686] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 2 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:38.650000 4702 raft_consensus.cc:2454] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 2.
I20240503 08:12:38.650476 4701 raft_consensus.cc:1203] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:38.655489 6009 consensus_queue.cc:237] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:38.661082 6078 ts_tablet_manager.cc:1398] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.023s user 0.003s sys 0.001s
I20240503 08:12:38.663844 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-266d0c75717b4fc98eae8b75a5ba4d8c received from {username='slave'} at 127.27.157.199:52311
I20240503 08:12:38.664057 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session f24648c92c384f1ba349eee801f985e2-266d0c75717b4fc98eae8b75a5ba4d8c on tablet 266d0c75717b4fc98eae8b75a5ba4d8c with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:12:38.670641 1667 catalog_manager.cc:5484] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 1 to 2, leader changed from 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:38.678136 4967 raft_consensus.cc:1203] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:38.683996 5948 consensus_queue.cc:1035] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:38.709643 5948 consensus_queue.cc:1035] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:38.709746 1667 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:38.726649 5983 consensus_queue.cc:1035] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:38.758481 4967 raft_consensus.cc:1261] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:38.759214 5919 consensus_queue.cc:1035] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:38.768775 4833 raft_consensus.cc:1261] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:38.769678 5919 consensus_queue.cc:1035] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:38.774654 1943 raft_consensus.cc:1203] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:38.785023 6019 consensus_queue.cc:1035] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:38.792042 1668 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet ad7cc4af7e954cb89b16ab7f518d24f8 (table rebalancer_dynamic_table [id=4906818a88ff460cbb11ce4f629fe500]). Expected version 1 got 0
I20240503 08:12:38.800518 5737 consensus_queue.cc:1035] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:38.802785 6113 tablet.cc:1722] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:38.809031 6113 tablet.cc:1620] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:38.810046 1668 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet ad7cc4af7e954cb89b16ab7f518d24f8 (table rebalancer_dynamic_table [id=4906818a88ff460cbb11ce4f629fe500]). Expected version 1 got 0
I20240503 08:12:38.815845 5786 consensus_queue.cc:1035] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:38.836983 6114 tablet.cc:1722] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:38.837545 6114 tablet.cc:1620] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:38.843169 1667 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet ad7cc4af7e954cb89b16ab7f518d24f8 (table rebalancer_dynamic_table [id=4906818a88ff460cbb11ce4f629fe500]). Expected version 1 got 0
I20240503 08:12:38.848301 1664 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=4906818a88ff460cbb11ce4f629fe500] alter complete (version 1)
I20240503 08:12:38.849234 6120 tablet.cc:1722] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:38.850500 1668 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet ad7cc4af7e954cb89b16ab7f518d24f8 (table rebalancer_dynamic_table [id=4906818a88ff460cbb11ce4f629fe500]). Expected version 1 got 0
I20240503 08:12:38.850869 6120 tablet.cc:1620] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:38.915078 1664 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:38.925890 4682 tablet_service.cc:1467] Processing CreateTablet for tablet b48714749469474886f9e1d7e7635915 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:38.926430 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48714749469474886f9e1d7e7635915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.927294 4946 tablet_service.cc:1467] Processing CreateTablet for tablet ebd7877c8faf409a8d042bef38f218aa (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:38.927397 1924 tablet_service.cc:1467] Processing CreateTablet for tablet b48714749469474886f9e1d7e7635915 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:38.927786 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd7877c8faf409a8d042bef38f218aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.927786 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48714749469474886f9e1d7e7635915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.931613 6153 tablet_bootstrap.cc:492] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:38.931826 4814 tablet_service.cc:1467] Processing CreateTablet for tablet f4a205bf277d4770836dbbf0ec28d884 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:38.932271 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a205bf277d4770836dbbf0ec28d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.933017 4683 tablet_service.cc:1467] Processing CreateTablet for tablet f4a205bf277d4770836dbbf0ec28d884 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:38.933477 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a205bf277d4770836dbbf0ec28d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.933684 4947 tablet_service.cc:1467] Processing CreateTablet for tablet f4a205bf277d4770836dbbf0ec28d884 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:38.933729 4681 tablet_service.cc:1467] Processing CreateTablet for tablet ebd7877c8faf409a8d042bef38f218aa (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:38.934135 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a205bf277d4770836dbbf0ec28d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.934196 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd7877c8faf409a8d042bef38f218aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.935073 6153 tablet_bootstrap.cc:654] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.940488 4568 raft_consensus.cc:1203] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:38.942693 4548 tablet_service.cc:1467] Processing CreateTablet for tablet b48714749469474886f9e1d7e7635915 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:38.943173 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48714749469474886f9e1d7e7635915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.942687 4547 tablet_service.cc:1467] Processing CreateTablet for tablet ebd7877c8faf409a8d042bef38f218aa (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:38.948084 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd7877c8faf409a8d042bef38f218aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:38.950278 6154 tablet_bootstrap.cc:492] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:38.952109 6154 tablet_bootstrap.cc:654] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.958556 6157 tablet_bootstrap.cc:492] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:38.958817 6156 tablet_bootstrap.cc:492] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:38.962363 6156 tablet_bootstrap.cc:654] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.963040 6157 tablet_bootstrap.cc:654] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.964268 1664 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:38.975102 6154 tablet_bootstrap.cc:492] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:38.975298 6154 ts_tablet_manager.cc:1367] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.005s
I20240503 08:12:38.976476 6154 raft_consensus.cc:348] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:38.977141 6154 raft_consensus.cc:374] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:38.977277 6154 raft_consensus.cc:729] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:38.977562 6154 consensus_queue.cc:260] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:38.978071 6113 tablet.cc:1722] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:38.978360 6154 ts_tablet_manager.cc:1398] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:38.978659 6113 tablet.cc:1620] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:38.979015 6154 tablet_bootstrap.cc:492] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:38.980633 6120 tablet.cc:1722] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:38.980981 6154 tablet_bootstrap.cc:654] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:38.981173 6120 tablet.cc:1620] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:38.982676 6114 tablet.cc:1722] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:38.983197 6114 tablet.cc:1620] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:38.991724 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=4906818a88ff460cbb11ce4f629fe500] alter complete (version 2)
I20240503 08:12:39.001798 6157 tablet_bootstrap.cc:492] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:39.002117 6157 ts_tablet_manager.cc:1367] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.005s
I20240503 08:12:39.004817 6157 raft_consensus.cc:348] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:39.005147 6157 raft_consensus.cc:374] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:39.005280 6157 raft_consensus.cc:729] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:39.005581 6157 consensus_queue.cc:260] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:39.006383 6157 ts_tablet_manager.cc:1398] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:39.007697 6158 tablet_bootstrap.cc:492] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:39.010339 6154 tablet_bootstrap.cc:492] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:39.013017 6158 tablet_bootstrap.cc:654] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:39.013058 6154 ts_tablet_manager.cc:1367] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.006s
I20240503 08:12:39.013950 6154 raft_consensus.cc:348] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.015417 6154 raft_consensus.cc:374] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:39.015702 6154 raft_consensus.cc:729] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:39.016322 6154 consensus_queue.cc:260] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.022337 6154 ts_tablet_manager.cc:1398] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 08:12:39.024793 6158 tablet_bootstrap.cc:492] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:39.026978 6158 ts_tablet_manager.cc:1367] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20240503 08:12:39.027590 6153 tablet_bootstrap.cc:492] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:39.027796 6153 ts_tablet_manager.cc:1367] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.096s user 0.021s sys 0.010s
I20240503 08:12:39.027992 6156 tablet_bootstrap.cc:492] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:39.028228 6156 ts_tablet_manager.cc:1367] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.070s user 0.013s sys 0.000s
I20240503 08:12:39.028625 6153 raft_consensus.cc:348] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:39.028890 6153 raft_consensus.cc:374] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:39.029016 6153 raft_consensus.cc:729] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:39.029291 6153 consensus_queue.cc:260] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:39.029325 6156 raft_consensus.cc:348] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:39.029664 6156 raft_consensus.cc:374] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:39.029830 6156 raft_consensus.cc:729] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:39.029920 6153 ts_tablet_manager.cc:1398] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:39.030263 6153 tablet_bootstrap.cc:492] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:39.030189 6156 consensus_queue.cc:260] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:39.032037 6153 tablet_bootstrap.cc:654] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:39.034235 6156 ts_tablet_manager.cc:1398] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:39.034024 6158 raft_consensus.cc:348] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.036059 6158 raft_consensus.cc:374] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:39.038477 4834 raft_consensus.cc:1203] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:39.042239 6156 tablet_bootstrap.cc:492] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:39.044461 6156 tablet_bootstrap.cc:654] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:39.052990 6156 tablet_bootstrap.cc:492] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:39.053315 6156 ts_tablet_manager.cc:1367] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 08:12:39.054354 6158 raft_consensus.cc:729] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:39.054323 6156 raft_consensus.cc:348] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:39.054944 6158 consensus_queue.cc:260] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.062364 6156 raft_consensus.cc:374] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:39.062951 6156 raft_consensus.cc:729] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:39.063663 6158 ts_tablet_manager.cc:1398] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.036s user 0.003s sys 0.000s
I20240503 08:12:39.063916 6156 consensus_queue.cc:260] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:39.066439 6156 ts_tablet_manager.cc:1398] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240503 08:12:39.068564 6153 tablet_bootstrap.cc:492] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:39.068778 6153 ts_tablet_manager.cc:1367] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.001s
I20240503 08:12:39.069656 6153 raft_consensus.cc:348] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:39.069891 6153 raft_consensus.cc:374] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:39.069970 6153 raft_consensus.cc:729] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:39.070246 6153 consensus_queue.cc:260] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:39.070849 6153 ts_tablet_manager.cc:1398] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:39.072281 6153 tablet_bootstrap.cc:492] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:39.074424 6153 tablet_bootstrap.cc:654] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:39.080929 5919 raft_consensus.cc:482] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:39.081229 6153 tablet_bootstrap.cc:492] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:39.081133 5919 raft_consensus.cc:504] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:39.081447 6153 ts_tablet_manager.cc:1367] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:12:39.081913 5919 leader_election.cc:288] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:39.082309 6153 raft_consensus.cc:348] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.082546 6153 raft_consensus.cc:374] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:39.082686 6153 raft_consensus.cc:729] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:39.082914 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48714749469474886f9e1d7e7635915" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:39.082984 6153 consensus_queue.cc:260] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.083200 1943 raft_consensus.cc:2454] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:39.083011 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48714749469474886f9e1d7e7635915" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:39.083528 4702 raft_consensus.cc:2454] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:39.083848 6153 ts_tablet_manager.cc:1398] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:39.084662 4504 leader_election.cc:302] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:39.084928 5919 raft_consensus.cc:2781] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:39.085057 5919 raft_consensus.cc:482] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:39.085160 5919 raft_consensus.cc:3037] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.087999 5919 raft_consensus.cc:504] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:39.088757 5919 leader_election.cc:288] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:39.089268 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48714749469474886f9e1d7e7635915" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:39.089483 4702 raft_consensus.cc:3037] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.089545 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48714749469474886f9e1d7e7635915" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:39.089807 1943 raft_consensus.cc:3037] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.092540 4702 raft_consensus.cc:2454] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:39.092592 1943 raft_consensus.cc:2454] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:39.093103 4503 leader_election.cc:302] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:39.093457 5919 raft_consensus.cc:2781] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:39.093650 5919 raft_consensus.cc:686] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:39.093964 5919 consensus_queue.cc:237] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:39.099459 1668 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:39.100855 1667 catalog_manager.cc:5484] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:39.119980 6113 tablet.cc:1722] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:39.128629 6120 tablet.cc:1722] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:39.131724 6114 tablet.cc:1722] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:39.143656 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=4906818a88ff460cbb11ce4f629fe500] alter complete (version 3)
I20240503 08:12:39.145150 6181 raft_consensus.cc:1050] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:39.146298 6181 consensus_queue.cc:237] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:39.161502 5417 raft_consensus.cc:1050] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:12:39.163538 5417 consensus_queue.cc:237] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:39.170622 1812 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:39.172371 6180 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:39.186486 4569 raft_consensus.cc:1261] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.187183 4967 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:39.187451 2201 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:39.190150 5863 consensus_queue.cc:1035] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:39.190507 6180 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:39.191568 5417 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:39.196585 2072 raft_consensus.cc:1261] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.200557 4967 raft_consensus.cc:1261] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.202065 6182 consensus_queue.cc:1035] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:39.203078 5417 consensus_queue.cc:1035] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:39.210906 2072 raft_consensus.cc:1261] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:12:39.211699 6009 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:39.214743 6180 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:39.218808 4702 raft_consensus.cc:1261] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:12:39.219578 6009 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:39.232916 6095 raft_consensus.cc:1050] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab: attempting to promote NON_VOTER 43e2e01cdd314f9c872c4336ef74250a to VOTER
I20240503 08:12:39.234850 6095 consensus_queue.cc:237] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:12:39.236205 1812 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:39.236821 2201 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:39.236995 1664 catalog_manager.cc:5484] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:39.243242 4967 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:39.248046 5875 raft_consensus.cc:1050] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:12:39.249243 5875 consensus_queue.cc:237] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:39.252724 1664 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:39.253038 1664 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:39.257292 1664 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet ad7cc4af7e954cb89b16ab7f518d24f8
I20240503 08:12:39.258699 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7cc4af7e954cb89b16ab7f518d24f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:39 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:39.259315 4569 raft_consensus.cc:2932] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.259904 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7cc4af7e954cb89b16ab7f518d24f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:39 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:39.259738 5863 raft_consensus.cc:2932] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.264436 4947 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7cc4af7e954cb89b16ab7f518d24f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:39 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:39.267120 1667 catalog_manager.cc:5484] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.268106 2073 raft_consensus.cc:2932] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.270639 1943 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:39.271373 2201 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:39.272163 5357 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:39.272717 4701 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:39.273991 5874 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:39.274699 5874 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:39.277472 1810 raft_consensus.cc:1261] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:12:39.278285 6009 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:39.282645 1942 consensus_queue.cc:237] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:39.278472 4964 raft_consensus.cc:2932] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.289400 4568 raft_consensus.cc:1261] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:39.290985 5863 consensus_queue.cc:1035] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:39.295795 5417 raft_consensus.cc:2932] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.302922 5874 raft_consensus.cc:1050] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a to VOTER
I20240503 08:12:39.303928 5874 consensus_queue.cc:237] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:12:39.306134 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4900ffdbc624376b32f054efdb100b5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:39.315892 6019 raft_consensus.cc:482] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:39.316124 6019 raft_consensus.cc:504] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.316932 6019 leader_election.cc:288] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:39.317853 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a205bf277d4770836dbbf0ec28d884" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:39.318153 4966 raft_consensus.cc:2454] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:39.318893 4768 leader_election.cc:302] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:39.319479 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a205bf277d4770836dbbf0ec28d884" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:39.319725 4703 raft_consensus.cc:2454] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:39.320235 6019 raft_consensus.cc:2781] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:39.320376 6019 raft_consensus.cc:482] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:39.320480 6019 raft_consensus.cc:3037] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.321933 1667 catalog_manager.cc:5484] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:39.323410 6019 raft_consensus.cc:504] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.324770 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a205bf277d4770836dbbf0ec28d884" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:39.324991 4703 raft_consensus.cc:3037] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.326416 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet f4900ffdbc624376b32f054efdb100b5 with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:39.328491 4703 raft_consensus.cc:2454] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:39.329272 4770 leader_election.cc:302] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:39.329730 5931 raft_consensus.cc:2781] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:39.329890 5931 raft_consensus.cc:686] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:39.330796 5931 consensus_queue.cc:237] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.334425 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a205bf277d4770836dbbf0ec28d884" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:39.334762 4966 raft_consensus.cc:3037] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.334522 5357 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.338004 4966 raft_consensus.cc:2454] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:39.339092 6019 leader_election.cc:288] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:39.342185 6187 raft_consensus.cc:482] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:39.342521 6201 tablet_replica.cc:330] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:39.342409 6187 raft_consensus.cc:504] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:39.342759 6201 raft_consensus.cc:2229] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:39.343034 6201 raft_consensus.cc:2258] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:39.346170 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7877c8faf409a8d042bef38f218aa" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:39.346437 4568 raft_consensus.cc:2454] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:39.346940 4635 leader_election.cc:302] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:39.347277 5948 raft_consensus.cc:2781] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:39.347483 5948 raft_consensus.cc:482] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:39.347590 5948 raft_consensus.cc:3037] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.348109 6187 leader_election.cc:288] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:39.348176 6199 tablet_replica.cc:330] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:39.348387 6199 raft_consensus.cc:2229] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:39.348440 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7877c8faf409a8d042bef38f218aa" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:39.348634 6199 raft_consensus.cc:2258] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:39.348671 4966 raft_consensus.cc:2454] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:39.349874 6199 ts_tablet_manager.cc:1875] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:39.350980 6200 tablet_replica.cc:330] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:39.351853 5948 raft_consensus.cc:504] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:39.351576 4568 raft_consensus.cc:2932] T f4900ffdbc624376b32f054efdb100b5 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.353456 5948 leader_election.cc:288] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:39.353545 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7877c8faf409a8d042bef38f218aa" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:39.353776 4569 raft_consensus.cc:3037] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.351275 6200 raft_consensus.cc:2229] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:39.357445 6200 raft_consensus.cc:2258] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:39.357776 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd7877c8faf409a8d042bef38f218aa" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:39.358012 4966 raft_consensus.cc:3037] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.358058 4569 raft_consensus.cc:2454] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:39.358505 6200 ts_tablet_manager.cc:1875] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:39.358669 4635 leader_election.cc:302] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:39.359028 5948 raft_consensus.cc:2781] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:39.359232 5948 raft_consensus.cc:686] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:39.360414 5948 consensus_queue.cc:237] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:39.360862 1667 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:39.368050 1943 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.369911 4966 raft_consensus.cc:2454] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:39.372789 2183 tablet_service.cc:1467] Processing CreateTablet for tablet fbf013dff097486eae19880827f3d3dd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1620d21223e442e68ee94438408a3dd2]), partition=
I20240503 08:12:39.373260 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf013dff097486eae19880827f3d3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:39.376303 4548 tablet_service.cc:1467] Processing CreateTablet for tablet fbf013dff097486eae19880827f3d3dd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1620d21223e442e68ee94438408a3dd2]), partition=
I20240503 08:12:39.376768 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf013dff097486eae19880827f3d3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:39.379479 6201 ts_tablet_manager.cc:1875] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:39.380972 4834 raft_consensus.cc:1261] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.381733 5919 consensus_queue.cc:1035] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:39.384406 4967 raft_consensus.cc:1261] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:39.388459 1812 raft_consensus.cc:1261] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.389153 4947 tablet_service.cc:1467] Processing CreateTablet for tablet fbf013dff097486eae19880827f3d3dd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1620d21223e442e68ee94438408a3dd2]), partition=
I20240503 08:12:39.390761 2202 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.396200 2072 raft_consensus.cc:1261] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.396646 5919 consensus_queue.cc:1035] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:39.396991 2201 consensus_queue.cc:237] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:39.397585 6094 consensus_queue.cc:1035] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 08:12:39.399371 4702 raft_consensus.cc:1261] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:39.400082 1810 raft_consensus.cc:1261] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:39.400090 6099 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:39.400434 1671 catalog_manager.cc:5484] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.400732 6099 consensus_queue.cc:1035] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:39.403847 4701 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.404704 1811 raft_consensus.cc:2932] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.403671 6212 raft_consensus.cc:2932] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.410356 1667 catalog_manager.cc:5484] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:39.411778 6222 tablet_bootstrap.cc:492] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:39.411815 4702 raft_consensus.cc:2932] T 1811bd652d2c4afb9f52bcd64ceb2acb P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.415009 6214 consensus_queue.cc:1035] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 08:12:39.422844 6222 tablet_bootstrap.cc:654] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:39.422982 1668 catalog_manager.cc:5484] T 1811bd652d2c4afb9f52bcd64ceb2acb P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 5 to 7, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 2 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.425441 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1811bd652d2c4afb9f52bcd64ceb2acb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:39.427486 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet 1811bd652d2c4afb9f52bcd64ceb2acb with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:39.429291 4964 raft_consensus.cc:2932] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.435145 6222 tablet_bootstrap.cc:492] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:39.435683 6222 ts_tablet_manager.cc:1367] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
I20240503 08:12:39.436843 6222 raft_consensus.cc:348] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.437191 6222 raft_consensus.cc:374] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:39.437355 6222 raft_consensus.cc:729] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:39.438467 1943 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.438494 4967 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.439419 5875 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:39.439985 4702 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.441125 6222 consensus_queue.cc:260] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.441287 5874 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:39.441794 6222 ts_tablet_manager.cc:1398] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:39.444492 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf013dff097486eae19880827f3d3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:39.450320 5357 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240503 08:12:39.453687 6154 tablet_bootstrap.cc:492] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:39.456172 6154 tablet_bootstrap.cc:654] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:39.465111 5919 raft_consensus.cc:2932] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:39.465904 6210 tablet_replica.cc:330] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:39.466141 6210 raft_consensus.cc:2229] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:39.466403 6210 raft_consensus.cc:2258] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:39.467684 6210 ts_tablet_manager.cc:1875] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:39.470046 2072 consensus_queue.cc:237] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:39.472707 4701 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:39.472891 1942 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:39.472901 1671 catalog_manager.cc:5484] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:39.473695 6201 ts_tablet_manager.cc:1888] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:39.473661 5357 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:39.473865 6201 log.cc:1196] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/ad7cc4af7e954cb89b16ab7f518d24f8
I20240503 08:12:39.474267 5357 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:39.474431 6201 ts_tablet_manager.cc:1909] T ad7cc4af7e954cb89b16ab7f518d24f8 P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:39.474416 2072 raft_consensus.cc:2932] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:39.477633 6154 tablet_bootstrap.cc:492] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:39.477948 6154 ts_tablet_manager.cc:1367] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.005s
I20240503 08:12:39.479995 6154 raft_consensus.cc:348] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.479981 5357 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.480341 6154 raft_consensus.cc:374] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:39.480549 6154 raft_consensus.cc:729] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:39.480846 6154 consensus_queue.cc:260] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.481554 6154 ts_tablet_manager.cc:1398] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:39.482502 4701 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.481377 1812 raft_consensus.cc:2932] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:39.487807 1942 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.489020 1665 catalog_manager.cc:5484] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:39.494593 6156 tablet_bootstrap.cc:492] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:39.493849 4834 raft_consensus.cc:2932] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:39.497017 6156 tablet_bootstrap.cc:654] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:39.502010 6200 ts_tablet_manager.cc:1888] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:39.503871 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet ad7cc4af7e954cb89b16ab7f518d24f8 (table rebalancer_dynamic_table_new_name [id=4906818a88ff460cbb11ce4f629fe500]) successfully deleted
I20240503 08:12:39.512763 5786 raft_consensus.cc:1050] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:39.513624 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet a92cb30e806b4e58a2d9f9cfa516bdf9 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:39.513722 5786 consensus_queue.cc:237] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:39.518349 1664 catalog_manager.cc:5484] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:39.521176 4703 raft_consensus.cc:1261] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.521966 5931 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:39.522162 6199 ts_tablet_manager.cc:1888] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:39.522361 6199 log.cc:1196] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/ad7cc4af7e954cb89b16ab7f518d24f8
I20240503 08:12:39.522979 6199 ts_tablet_manager.cc:1909] T ad7cc4af7e954cb89b16ab7f518d24f8 P f24648c92c384f1ba349eee801f985e2: Deleting consensus metadata
I20240503 08:12:39.524495 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet ad7cc4af7e954cb89b16ab7f518d24f8 (table rebalancer_dynamic_table_new_name [id=4906818a88ff460cbb11ce4f629fe500]) successfully deleted
I20240503 08:12:39.526407 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a92cb30e806b4e58a2d9f9cfa516bdf9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:39.530200 6218 raft_consensus.cc:1050] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:12:39.532878 1812 raft_consensus.cc:1261] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.533701 5931 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:39.531138 6218 consensus_queue.cc:237] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:39.536136 6210 ts_tablet_manager.cc:1888] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:39.536367 6210 log.cc:1196] T f4900ffdbc624376b32f054efdb100b5 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/f4900ffdbc624376b32f054efdb100b5
I20240503 08:12:39.537324 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet f4900ffdbc624376b32f054efdb100b5 (table rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]) successfully deleted
I20240503 08:12:39.537806 6210 tablet_replica.cc:330] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:39.538106 6210 raft_consensus.cc:2229] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:39.538412 6210 raft_consensus.cc:2258] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:39.545610 4702 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:39.541769 5874 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:39.563310 1942 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:39.564590 5786 raft_consensus.cc:2932] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.565620 6210 ts_tablet_manager.cc:1875] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:39.568732 6200 log.cc:1196] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/ad7cc4af7e954cb89b16ab7f518d24f8
I20240503 08:12:39.569320 6200 ts_tablet_manager.cc:1909] T ad7cc4af7e954cb89b16ab7f518d24f8 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:39.568945 4964 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:39.570989 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet ad7cc4af7e954cb89b16ab7f518d24f8 (table rebalancer_dynamic_table_new_name [id=4906818a88ff460cbb11ce4f629fe500]) successfully deleted
I20240503 08:12:39.507308 6156 tablet_bootstrap.cc:492] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:39.571542 6156 ts_tablet_manager.cc:1367] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.077s user 0.008s sys 0.000s
I20240503 08:12:39.572171 1671 catalog_manager.cc:5484] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:39.572535 6156 raft_consensus.cc:348] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.572743 6156 raft_consensus.cc:374] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:39.572829 6156 raft_consensus.cc:729] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:39.573060 6156 consensus_queue.cc:260] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.573608 6156 ts_tablet_manager.cc:1398] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:39.512787 4568 consensus_queue.cc:237] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:12:39.520114 4569 raft_consensus.cc:1261] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:39.576049 4835 raft_consensus.cc:1261] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:39.577674 2201 raft_consensus.cc:1261] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:39.577651 5874 consensus_queue.cc:1035] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:39.578294 6202 consensus_queue.cc:1035] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:39.580621 1810 raft_consensus.cc:2932] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.584515 4834 raft_consensus.cc:1261] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:39.586228 6094 consensus_queue.cc:1035] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:39.531996 6219 consensus_queue.cc:1035] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:39.587878 6019 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:39.597080 1665 catalog_manager.cc:5484] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.601848 6095 raft_consensus.cc:2932] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:39.607484 6219 raft_consensus.cc:482] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:39.607743 6219 raft_consensus.cc:504] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.608023 4834 raft_consensus.cc:2932] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:39.608848 6219 leader_election.cc:288] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:39.618090 6210 ts_tablet_manager.cc:1888] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:12:39.624562 4701 raft_consensus.cc:2932] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.628762 1656 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3cda734f360c4d3eac88a6517028e49d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:39.629627 1667 catalog_manager.cc:5484] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:39.634318 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cda734f360c4d3eac88a6517028e49d with delete_type TABLET_DATA_TOMBSTONED (TS b8f581390da449ccb9e299011296405e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:39.638706 1943 raft_consensus.cc:1261] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:39.652053 6212 raft_consensus.cc:482] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:39.652309 6212 raft_consensus.cc:504] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.653262 6212 leader_election.cc:288] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:39.653358 6247 tablet_replica.cc:330] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:39.653604 6247 raft_consensus.cc:2229] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:39.653915 6247 raft_consensus.cc:2258] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:39.656889 4964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf013dff097486eae19880827f3d3dd" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:39.657174 4964 raft_consensus.cc:2454] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:39.657701 6234 tablet_replica.cc:330] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:39.657953 6234 raft_consensus.cc:2229] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:39.658288 6234 raft_consensus.cc:2258] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:39.659452 6234 ts_tablet_manager.cc:1875] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:39.661577 6247 ts_tablet_manager.cc:1875] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:39.664094 4502 leader_election.cc:302] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:39.665326 6219 raft_consensus.cc:2781] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:39.665448 6219 raft_consensus.cc:482] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:39.665586 6219 raft_consensus.cc:3037] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.668733 6219 raft_consensus.cc:504] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.669987 6219 leader_election.cc:288] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:39.673527 4964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf013dff097486eae19880827f3d3dd" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:39.674468 4964 raft_consensus.cc:3037] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.633814 2072 raft_consensus.cc:1261] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:39.639948 5874 consensus_queue.cc:1035] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:39.640578 2074 consensus_queue.cc:237] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:12:39.678526 4964 raft_consensus.cc:2454] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:39.679347 4502 leader_election.cc:302] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:39.680713 6219 raft_consensus.cc:2781] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:39.681053 6219 raft_consensus.cc:686] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:39.681484 6219 consensus_queue.cc:237] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:39.682335 1944 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:39.683043 5874 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:39.686517 5874 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:39.687760 6247 ts_tablet_manager.cc:1888] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:39.687906 6247 log.cc:1196] T 3cda734f360c4d3eac88a6517028e49d P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/3cda734f360c4d3eac88a6517028e49d
I20240503 08:12:39.688880 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 3cda734f360c4d3eac88a6517028e49d (table rebalancer_extra_table_10 [id=a4efcdccd12c4091b793c47928fe4764]) successfully deleted
I20240503 08:12:39.688652 1665 catalog_manager.cc:5484] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:39.690016 1944 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:39.691280 4569 raft_consensus.cc:2932] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:39.698246 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:39.703125 6219 consensus_queue.cc:1035] T 3cda734f360c4d3eac88a6517028e49d P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:39.704826 1665 catalog_manager.cc:5484] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:39.705511 2072 raft_consensus.cc:2932] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
W20240503 08:12:39.655999 6238 log.cc:924] Time spent T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20240503 08:12:39.709700 4947 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 with delete_type TABLET_DATA_TOMBSTONED (TS 00b61625d8a7460892c118a2570afda5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:39.710078 6201 tablet_replica.cc:330] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:39.710345 6201 raft_consensus.cc:2229] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:39.710575 6201 raft_consensus.cc:2258] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:39.711812 6201 ts_tablet_manager.cc:1875] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:39.709951 6218 raft_consensus.cc:2932] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:39.720206 6201 ts_tablet_manager.cc:1888] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:39.720440 6201 log.cc:1196] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/2d4d06fdfc4e49d5b0790e2ee3b4cc83
I20240503 08:12:39.721382 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 (table rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]) successfully deleted
I20240503 08:12:39.744117 6210 log.cc:1196] T 1811bd652d2c4afb9f52bcd64ceb2acb P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/1811bd652d2c4afb9f52bcd64ceb2acb
I20240503 08:12:39.745043 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf013dff097486eae19880827f3d3dd" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:39.747121 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 1811bd652d2c4afb9f52bcd64ceb2acb (table rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]) successfully deleted
I20240503 08:12:39.756608 4701 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:39.757032 1943 raft_consensus.cc:2932] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:39.758276 6219 consensus_queue.cc:1035] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:39.759531 5874 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:39.761605 4703 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:39.767746 1664 catalog_manager.cc:5484] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:39.774780 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf013dff097486eae19880827f3d3dd" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:39.774942 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf013dff097486eae19880827f3d3dd" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:39.775097 2201 raft_consensus.cc:2454] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:39.777160 2201 raft_consensus.cc:2932] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:39.778379 4835 raft_consensus.cc:2932] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:39.782313 4964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf013dff097486eae19880827f3d3dd" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:39.782677 4964 raft_consensus.cc:2379] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 184c543cb05d49cc98ba1c56617be7a5 in current term 1: Already voted for candidate 5f95b05d111e4090bbc59dec394ed0ab in this term.
I20240503 08:12:39.783452 2138 leader_election.cc:302] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5; no voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:39.784415 6212 raft_consensus.cc:3037] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:39.787817 6212 raft_consensus.cc:2726] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:39.795971 5931 consensus_queue.cc:1035] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:39.806540 2072 consensus_queue.cc:237] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:39.812976 1942 raft_consensus.cc:1261] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:39.813043 4835 raft_consensus.cc:1261] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:39.813776 5874 consensus_queue.cc:1035] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:39.816684 5874 consensus_queue.cc:1035] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:39.818356 5875 raft_consensus.cc:2932] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:39.823756 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 266d0c75717b4fc98eae8b75a5ba4d8c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:39.825950 6234 ts_tablet_manager.cc:1888] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:39.826134 6234 log.cc:1196] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/a92cb30e806b4e58a2d9f9cfa516bdf9
I20240503 08:12:39.827767 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet a92cb30e806b4e58a2d9f9cfa516bdf9 (table rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]) successfully deleted
I20240503 08:12:39.831808 1942 raft_consensus.cc:2932] T 266d0c75717b4fc98eae8b75a5ba4d8c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:39.837618 1665 catalog_manager.cc:5484] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:39.842072 4834 raft_consensus.cc:2932] T 266d0c75717b4fc98eae8b75a5ba4d8c P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:39.844636 1671 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:39.853873 6019 consensus_queue.cc:1035] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:39.858695 2201 raft_consensus.cc:1261] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:39.860209 4967 raft_consensus.cc:1261] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:39.860975 5919 consensus_queue.cc:1035] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:39.862166 5919 consensus_queue.cc:1035] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:39.862339 5948 consensus_queue.cc:1035] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:39.871362 1671 catalog_manager.cc:5576] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183) does not have the latest schema for tablet fbf013dff097486eae19880827f3d3dd (table rebalancer_dynamic_table [id=1620d21223e442e68ee94438408a3dd2]). Expected version 1 got 0
I20240503 08:12:39.877936 6208 mvcc.cc:205] Tried to move back new op lower bound from 7023509339573821440 to 7023509338859757568. Current Snapshot: MvccSnapshot[applied={T|T < 7023509339573821440}]
I20240503 08:12:39.889870 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 266d0c75717b4fc98eae8b75a5ba4d8c with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:39.890384 6234 tablet_replica.cc:330] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:39.890602 6234 raft_consensus.cc:2229] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:39.890859 6234 raft_consensus.cc:2258] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:39.891665 6267 tablet.cc:1722] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:39.892531 6267 tablet.cc:1620] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:39.892741 6234 ts_tablet_manager.cc:1875] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:39.900585 6234 ts_tablet_manager.cc:1888] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:39.900760 6234 log.cc:1196] T 266d0c75717b4fc98eae8b75a5ba4d8c P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/266d0c75717b4fc98eae8b75a5ba4d8c
I20240503 08:12:39.901626 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 266d0c75717b4fc98eae8b75a5ba4d8c (table rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]) successfully deleted
I20240503 08:12:39.904060 6265 tablet.cc:1722] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:39.904865 6266 tablet.cc:1722] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:39.908417 1667 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet fbf013dff097486eae19880827f3d3dd (table rebalancer_dynamic_table [id=1620d21223e442e68ee94438408a3dd2]). Expected version 1 got 0
I20240503 08:12:39.916509 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=1620d21223e442e68ee94438408a3dd2] alter complete (version 1)
I20240503 08:12:39.917235 1665 catalog_manager.cc:5576] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183) does not have the latest schema for tablet fbf013dff097486eae19880827f3d3dd (table rebalancer_dynamic_table [id=1620d21223e442e68ee94438408a3dd2]). Expected version 1 got 0
I20240503 08:12:39.918287 1671 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet fbf013dff097486eae19880827f3d3dd (table rebalancer_dynamic_table [id=1620d21223e442e68ee94438408a3dd2]). Expected version 1 got 0
I20240503 08:12:39.922992 6271 consensus_queue.cc:1035] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:39.926142 6265 tablet.cc:1620] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:39.928539 6266 tablet.cc:1620] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: MemRowSet was empty: no flush needed.
I20240503 08:12:40.014081 4835 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6cff57835b846e6b48c08bef52f5b68"
dest_uuid: "f24648c92c384f1ba349eee801f985e2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40526
I20240503 08:12:40.014350 4835 raft_consensus.cc:595] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:40.028812 1667 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:40.031463 2201 consensus_queue.cc:237] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:40.036336 2072 raft_consensus.cc:1261] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:40.037395 6301 consensus_queue.cc:1035] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:40.037784 1942 raft_consensus.cc:1261] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:40.038213 2140 consensus_peers.cc:485] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 314d235cdca241f8b80a7046fb1f524c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:40.039860 6225 consensus_queue.cc:1035] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:40.039983 6265 tablet.cc:1722] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:40.040556 6265 tablet.cc:1620] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:40.041391 6267 tablet.cc:1722] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:40.041616 6266 tablet.cc:1722] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:40.041885 6267 tablet.cc:1620] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:40.042214 6266 tablet.cc:1620] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: MemRowSet was empty: no flush needed.
I20240503 08:12:40.045687 6212 raft_consensus.cc:2932] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:40.047855 1942 raft_consensus.cc:2932] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:40.056460 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:40.064402 1667 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=1620d21223e442e68ee94438408a3dd2] alter complete (version 2)
I20240503 08:12:40.069528 1671 catalog_manager.cc:5484] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:40.076797 5786 raft_consensus.cc:979] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: : Instructing follower 31f41542452b4b1eb994ff4223d039fc to start an election
I20240503 08:12:40.077014 5786 raft_consensus.cc:1067] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Signalling peer 31f41542452b4b1eb994ff4223d039fc to start an election
I20240503 08:12:40.079160 4700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6cff57835b846e6b48c08bef52f5b68"
dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
from {username='slave'} at 127.27.157.199:49041
I20240503 08:12:40.080803 4700 raft_consensus.cc:482] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:40.080977 4700 raft_consensus.cc:3037] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:40.083747 2072 raft_consensus.cc:2932] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:40.085620 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8a48fca96c0e4fdfa7cd9f966b583e55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.086113 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.090099 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8a48fca96c0e4fdfa7cd9f966b583e55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.090484 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.095845 4835 consensus_queue.cc:237] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:40.096493 4947 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8a48fca96c0e4fdfa7cd9f966b583e55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.097067 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.103569 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 929d8d354b914502b5e677ccc84e2eda (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8a48fca96c0e4fdfa7cd9f966b583e55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.104108 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929d8d354b914502b5e677ccc84e2eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.104084 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 7f79a778fd444235a26fff1b713d6aac (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8a48fca96c0e4fdfa7cd9f966b583e55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.104679 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f79a778fd444235a26fff1b713d6aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.108505 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 7f79a778fd444235a26fff1b713d6aac (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8a48fca96c0e4fdfa7cd9f966b583e55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
W20240503 08:12:40.108678 4768 consensus_peers.cc:485] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54fe406b5f324d629243f7f939535620. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:40.109045 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f79a778fd444235a26fff1b713d6aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.109628 4966 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:40.110292 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 929d8d354b914502b5e677ccc84e2eda (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8a48fca96c0e4fdfa7cd9f966b583e55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.110759 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929d8d354b914502b5e677ccc84e2eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.114743 6311 tablet_bootstrap.cc:492] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:40.115833 4813 tablet_service.cc:1467] Processing CreateTablet for tablet 929d8d354b914502b5e677ccc84e2eda (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8a48fca96c0e4fdfa7cd9f966b583e55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.116343 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929d8d354b914502b5e677ccc84e2eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.116642 6311 tablet_bootstrap.cc:654] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.116536 6019 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:40.119570 2072 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:40.120458 5786 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:40.127933 5786 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:40.129215 6316 tablet_bootstrap.cc:492] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:40.131654 6316 tablet_bootstrap.cc:654] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.131810 2074 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:40.134100 6311 tablet_bootstrap.cc:492] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:40.134326 6311 ts_tablet_manager.cc:1367] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.010s
I20240503 08:12:40.135538 6311 raft_consensus.cc:348] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:40.135926 6311 raft_consensus.cc:374] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.136071 6311 raft_consensus.cc:729] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.136402 6311 consensus_queue.cc:260] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:40.137045 6311 ts_tablet_manager.cc:1398] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:40.138002 6313 tablet_bootstrap.cc:492] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:40.138425 6311 tablet_bootstrap.cc:492] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:40.140466 6311 tablet_bootstrap.cc:654] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.140269 1672 catalog_manager.cc:5484] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:40.140965 6313 tablet_bootstrap.cc:654] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.150457 6313 tablet_bootstrap.cc:492] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:40.150748 6313 ts_tablet_manager.cc:1367] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240503 08:12:40.151706 6313 raft_consensus.cc:348] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:40.152021 6313 raft_consensus.cc:374] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.152158 6313 raft_consensus.cc:729] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.152470 6313 consensus_queue.cc:260] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:40.153712 6313 ts_tablet_manager.cc:1398] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:40.161736 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 7f79a778fd444235a26fff1b713d6aac (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=8a48fca96c0e4fdfa7cd9f966b583e55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:40.162230 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f79a778fd444235a26fff1b713d6aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.135566 4700 raft_consensus.cc:504] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:40.164222 5874 raft_consensus.cc:482] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:40.164443 5874 raft_consensus.cc:504] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:40.165136 5874 leader_election.cc:288] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:40.166575 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cff57835b846e6b48c08bef52f5b68" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:40.166821 4835 raft_consensus.cc:3032] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:40.166899 4835 raft_consensus.cc:729] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:40.167245 4835 consensus_queue.cc:260] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:40.167706 4835 raft_consensus.cc:3037] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:12:40.170404 2010 leader_election.cc:341] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Not found: Tablet not found: 929d8d354b914502b5e677ccc84e2eda
I20240503 08:12:40.171454 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929d8d354b914502b5e677ccc84e2eda" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
W20240503 08:12:40.175271 2010 leader_election.cc:341] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:40.175482 2010 leader_election.cc:302] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a; no voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2
I20240503 08:12:40.175144 6311 tablet_bootstrap.cc:492] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:40.175853 6218 raft_consensus.cc:2726] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:40.176070 6311 ts_tablet_manager.cc:1367] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.018s
I20240503 08:12:40.177310 6311 raft_consensus.cc:348] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:40.179006 6311 raft_consensus.cc:374] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.179383 6311 raft_consensus.cc:729] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.179514 1668 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:40.180071 6311 consensus_queue.cc:260] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:40.182267 6311 ts_tablet_manager.cc:1398] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:12:40.186455 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cff57835b846e6b48c08bef52f5b68" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:40.186826 1812 raft_consensus.cc:3037] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:40.197690 4835 raft_consensus.cc:2454] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 2.
I20240503 08:12:40.204665 1812 raft_consensus.cc:2454] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 2.
I20240503 08:12:40.204818 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cff57835b846e6b48c08bef52f5b68" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:40.205163 4569 raft_consensus.cc:3037] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:40.208313 4700 leader_election.cc:288] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:40.213836 6265 tablet.cc:1722] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:40.222122 6267 tablet.cc:1722] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:40.232196 4638 leader_election.cc:302] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:40.232553 6271 raft_consensus.cc:2781] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:40.232748 6271 raft_consensus.cc:686] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 2 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:40.233047 6271 consensus_queue.cc:237] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:40.236308 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=1620d21223e442e68ee94438408a3dd2] alter complete (version 3)
I20240503 08:12:40.237601 1667 catalog_manager.cc:5484] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 1 to 2, leader changed from f24648c92c384f1ba349eee801f985e2 (127.27.157.199) to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 2 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:40.251693 4569 raft_consensus.cc:2454] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 2.
I20240503 08:12:40.253126 6266 tablet.cc:1722] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:40.256697 6326 tablet_bootstrap.cc:492] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:40.258980 6326 tablet_bootstrap.cc:654] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.259722 2073 consensus_queue.cc:237] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:40.265760 6313 tablet_bootstrap.cc:492] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:40.265990 4965 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:40.268339 6313 tablet_bootstrap.cc:654] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.272619 1942 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:40.273854 5875 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:40.279254 6327 tablet_bootstrap.cc:492] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:40.280386 6323 tablet_bootstrap.cc:492] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:40.282156 6327 tablet_bootstrap.cc:654] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
W20240503 08:12:40.282335 2010 consensus_peers.cc:485] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 044d8e5a69444865980ffc06a6be9080. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:40.283121 6323 tablet_bootstrap.cc:654] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.283149 2201 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:40.284686 6218 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:40.285637 6313 tablet_bootstrap.cc:492] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:40.285868 6313 ts_tablet_manager.cc:1367] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20240503 08:12:40.286330 6326 tablet_bootstrap.cc:492] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:40.286855 6313 raft_consensus.cc:348] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:40.287254 6316 tablet_bootstrap.cc:492] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:40.287274 6313 raft_consensus.cc:374] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.287387 6313 raft_consensus.cc:729] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.287600 6316 ts_tablet_manager.cc:1367] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.158s user 0.008s sys 0.000s
I20240503 08:12:40.287772 6313 consensus_queue.cc:260] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:40.289580 6313 ts_tablet_manager.cc:1398] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:40.289129 6316 raft_consensus.cc:348] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:40.290060 6316 raft_consensus.cc:374] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.290359 6316 raft_consensus.cc:729] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.290715 6316 consensus_queue.cc:260] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:40.291489 6326 ts_tablet_manager.cc:1367] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.003s
I20240503 08:12:40.292084 6316 ts_tablet_manager.cc:1398] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:12:40.293677 6326 raft_consensus.cc:348] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:40.294463 6316 tablet_bootstrap.cc:492] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:40.295790 6327 tablet_bootstrap.cc:492] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:40.296043 6327 ts_tablet_manager.cc:1367] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20240503 08:12:40.296818 6316 tablet_bootstrap.cc:654] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.297099 6327 raft_consensus.cc:348] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:40.297430 6327 raft_consensus.cc:374] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.297595 6327 raft_consensus.cc:729] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.298017 6327 consensus_queue.cc:260] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:40.298774 6327 ts_tablet_manager.cc:1398] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:40.300633 6326 raft_consensus.cc:374] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.300899 6326 raft_consensus.cc:729] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.301903 6326 consensus_queue.cc:260] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:40.302645 5875 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:40.304814 6326 ts_tablet_manager.cc:1398] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20240503 08:12:40.306324 2204 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:40.314393 1668 catalog_manager.cc:5484] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:40.309208 6316 tablet_bootstrap.cc:492] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:40.317289 1942 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:40.319561 6323 tablet_bootstrap.cc:492] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:40.319818 6323 ts_tablet_manager.cc:1367] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.000s
I20240503 08:12:40.321215 6323 raft_consensus.cc:348] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:40.321878 6323 raft_consensus.cc:374] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.322042 6323 raft_consensus.cc:729] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.322546 6323 consensus_queue.cc:260] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:40.324443 6323 ts_tablet_manager.cc:1398] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:40.317752 6316 ts_tablet_manager.cc:1367] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.005s
I20240503 08:12:40.325716 6316 raft_consensus.cc:348] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:40.332528 5875 raft_consensus.cc:482] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:40.332743 5875 raft_consensus.cc:504] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:40.333599 5875 leader_election.cc:288] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:40.333920 6316 raft_consensus.cc:374] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.334122 6316 raft_consensus.cc:729] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.334465 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3aaa1025ca4b0e832535d1ca3d5c4f" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:40.334734 4966 raft_consensus.cc:2454] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:40.335395 2008 leader_election.cc:302] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:40.335826 5875 raft_consensus.cc:2781] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:40.335968 5875 raft_consensus.cc:482] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:40.336072 5875 raft_consensus.cc:3037] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.336055 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3aaa1025ca4b0e832535d1ca3d5c4f" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:40.339494 5786 raft_consensus.cc:482] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:40.339829 5786 raft_consensus.cc:504] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:40.340117 5875 raft_consensus.cc:504] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:40.341256 5786 leader_election.cc:288] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:40.341291 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3aaa1025ca4b0e832535d1ca3d5c4f" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:40.341452 5875 leader_election.cc:288] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:40.341522 4966 raft_consensus.cc:3037] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.341922 6316 consensus_queue.cc:260] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:40.342370 4835 raft_consensus.cc:2454] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:40.342484 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3aaa1025ca4b0e832535d1ca3d5c4f" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:40.342842 4834 raft_consensus.cc:3037] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.343885 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929d8d354b914502b5e677ccc84e2eda" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:40.344244 2073 raft_consensus.cc:2454] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:40.344965 4768 leader_election.cc:302] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:40.345460 5931 raft_consensus.cc:2781] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:40.345571 5931 raft_consensus.cc:482] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:40.345680 5931 raft_consensus.cc:3037] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.345971 4966 raft_consensus.cc:2454] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:40.346423 2008 leader_election.cc:302] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:40.346799 5875 raft_consensus.cc:2781] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:40.346952 5875 raft_consensus.cc:686] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:40.347258 5875 consensus_queue.cc:237] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:40.347846 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929d8d354b914502b5e677ccc84e2eda" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:40.348107 4700 raft_consensus.cc:2454] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:40.348752 4834 raft_consensus.cc:2454] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:40.350171 6316 ts_tablet_manager.cc:1398] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.025s user 0.000s sys 0.003s
I20240503 08:12:40.351154 5931 raft_consensus.cc:504] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:40.352589 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929d8d354b914502b5e677ccc84e2eda" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:40.352856 4700 raft_consensus.cc:3037] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.355537 5931 leader_election.cc:288] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:40.356220 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929d8d354b914502b5e677ccc84e2eda" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:40.356472 2073 raft_consensus.cc:3037] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.357427 4700 raft_consensus.cc:2454] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:40.358022 4770 leader_election.cc:302] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:40.358387 5931 raft_consensus.cc:2781] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:40.358543 5931 raft_consensus.cc:686] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:40.358810 5931 consensus_queue.cc:237] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:40.358857 1668 catalog_manager.cc:5484] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:40.360435 2073 raft_consensus.cc:2454] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:40.365659 1668 catalog_manager.cc:5484] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:40.366151 1667 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:40.366616 1667 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:40.368994 4834 consensus_queue.cc:237] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:40.370726 1667 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet fbf013dff097486eae19880827f3d3dd
I20240503 08:12:40.372159 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf013dff097486eae19880827f3d3dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:40.372474 6234 tablet_replica.cc:330] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:40.372690 6234 raft_consensus.cc:2229] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:40.372910 6234 raft_consensus.cc:2258] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.374516 1942 raft_consensus.cc:1261] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:40.374540 4569 raft_consensus.cc:1261] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:40.375195 6019 consensus_queue.cc:1035] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:40.376044 5896 raft_consensus.cc:482] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:40.376101 6234 ts_tablet_manager.cc:1875] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.376184 5896 raft_consensus.cc:504] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:40.376781 5896 leader_election.cc:288] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:40.377470 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f79a778fd444235a26fff1b713d6aac" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:40.377707 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f79a778fd444235a26fff1b713d6aac" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:40.377802 2201 raft_consensus.cc:2454] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:40.377836 5931 consensus_queue.cc:1035] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:40.377955 4569 raft_consensus.cc:2454] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:40.378422 4900 leader_election.cc:302] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
W20240503 08:12:40.379631 4768 consensus_peers.cc:485] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f161c8da3b0e4b12b85ca0420a86dc07. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:40.380139 4946 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf013dff097486eae19880827f3d3dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:40.380163 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf013dff097486eae19880827f3d3dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:40 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:40.380493 5896 raft_consensus.cc:2781] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:40.380662 5896 raft_consensus.cc:482] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:40.380793 5896 raft_consensus.cc:3037] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.385133 5896 raft_consensus.cc:504] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:40.390705 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f79a778fd444235a26fff1b713d6aac" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:40.390954 4568 raft_consensus.cc:3037] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.394027 4568 raft_consensus.cc:2454] T 7f79a778fd444235a26fff1b713d6aac P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:40.394973 4900 leader_election.cc:302] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:40.395296 5983 raft_consensus.cc:2781] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:40.398716 5896 leader_election.cc:288] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:40.402710 6019 raft_consensus.cc:2932] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:40.404666 6344 tablet_replica.cc:330] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:40.404897 6344 raft_consensus.cc:2229] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:40.405115 6344 raft_consensus.cc:2258] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.405702 6345 tablet_replica.cc:330] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:40.405691 6234 ts_tablet_manager.cc:1888] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:40.405840 6234 log.cc:1196] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/fbf013dff097486eae19880827f3d3dd
I20240503 08:12:40.405922 6345 raft_consensus.cc:2229] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:40.406299 6345 raft_consensus.cc:2258] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:40.406476 6234 ts_tablet_manager.cc:1909] T fbf013dff097486eae19880827f3d3dd P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:40.408174 6345 ts_tablet_manager.cc:1875] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.408447 6344 ts_tablet_manager.cc:1875] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:40.411507 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet fbf013dff097486eae19880827f3d3dd (table rebalancer_dynamic_table_new_name [id=1620d21223e442e68ee94438408a3dd2]) successfully deleted
I20240503 08:12:40.413168 4568 raft_consensus.cc:2932] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:40.416791 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f79a778fd444235a26fff1b713d6aac" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:40.417121 2201 raft_consensus.cc:3037] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.416994 1668 catalog_manager.cc:5484] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:40.425899 2201 raft_consensus.cc:2454] T 7f79a778fd444235a26fff1b713d6aac P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:40.429538 1942 raft_consensus.cc:2932] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:40.439630 6350 ts_tablet_manager.cc:902] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:40.452911 5983 raft_consensus.cc:686] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:40.455252 6345 ts_tablet_manager.cc:1888] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:40.455787 6344 ts_tablet_manager.cc:1888] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:40.456014 6344 log.cc:1196] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/fbf013dff097486eae19880827f3d3dd
I20240503 08:12:40.453382 5983 consensus_queue.cc:237] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:40.457368 6344 ts_tablet_manager.cc:1909] T fbf013dff097486eae19880827f3d3dd P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:40.457571 6345 log.cc:1196] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/fbf013dff097486eae19880827f3d3dd
I20240503 08:12:40.459412 6345 ts_tablet_manager.cc:1909] T fbf013dff097486eae19880827f3d3dd P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:40.459416 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet fbf013dff097486eae19880827f3d3dd (table rebalancer_dynamic_table_new_name [id=1620d21223e442e68ee94438408a3dd2]) successfully deleted
I20240503 08:12:40.460909 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet fbf013dff097486eae19880827f3d3dd (table rebalancer_dynamic_table_new_name [id=1620d21223e442e68ee94438408a3dd2]) successfully deleted
I20240503 08:12:40.463603 6350 tablet_copy_client.cc:308] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.196:42183
I20240503 08:12:40.464565 2224 tablet_copy_service.cc:140] P 184c543cb05d49cc98ba1c56617be7a5: Received BeginTabletCopySession request for tablet 314d235cdca241f8b80a7046fb1f524c from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:51901)
I20240503 08:12:40.464766 2224 tablet_copy_service.cc:161] P 184c543cb05d49cc98ba1c56617be7a5: Beginning new tablet copy session on tablet 314d235cdca241f8b80a7046fb1f524c from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:51901: session id = 31f41542452b4b1eb994ff4223d039fc-314d235cdca241f8b80a7046fb1f524c
I20240503 08:12:40.467389 2224 tablet_copy_source_session.cc:205] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:40.467598 1667 catalog_manager.cc:5484] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:40.468993 6350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314d235cdca241f8b80a7046fb1f524c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.476963 6350 tablet_copy_client.cc:783] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:40.477411 6350 tablet_copy_client.cc:647] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:40.479291 2073 consensus_queue.cc:237] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:40.479900 6350 tablet_copy_client.cc:523] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:40.483004 2201 raft_consensus.cc:1261] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:40.484899 6350 tablet_bootstrap.cc:492] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
W20240503 08:12:40.485092 2010 consensus_peers.cc:485] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 133911d44b394e24a89a5d9443230d32. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:40.486678 5875 consensus_queue.cc:1035] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:40.487556 1942 raft_consensus.cc:1261] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:40.488318 5875 consensus_queue.cc:1035] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:40.490257 1667 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:40.499043 6218 raft_consensus.cc:2932] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:40.501439 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb5ca2c4a6124ffaa9fde8ae5a10494a]), partition=
I20240503 08:12:40.501899 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.504552 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb5ca2c4a6124ffaa9fde8ae5a10494a]), partition=
I20240503 08:12:40.504551 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb5ca2c4a6124ffaa9fde8ae5a10494a]), partition=
I20240503 08:12:40.504941 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.504949 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.508873 1668 catalog_manager.cc:5484] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:40.511497 6311 tablet_bootstrap.cc:492] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:40.511870 2201 raft_consensus.cc:2932] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:40.513861 6311 tablet_bootstrap.cc:654] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.516162 6323 tablet_bootstrap.cc:492] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:40.518903 6323 tablet_bootstrap.cc:654] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.526139 6326 tablet_bootstrap.cc:492] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:40.528622 6326 tablet_bootstrap.cc:654] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:40.528335 1942 raft_consensus.cc:2932] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:40.539848 6323 tablet_bootstrap.cc:492] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:40.540163 6323 ts_tablet_manager.cc:1367] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240503 08:12:40.541167 6323 raft_consensus.cc:348] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:40.541718 6323 raft_consensus.cc:374] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.541808 6323 raft_consensus.cc:729] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.542116 6323 consensus_queue.cc:260] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:40.546774 6323 ts_tablet_manager.cc:1398] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:40.570309 6311 tablet_bootstrap.cc:492] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:40.570711 6311 ts_tablet_manager.cc:1367] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.059s user 0.006s sys 0.003s
I20240503 08:12:40.570950 6326 tablet_bootstrap.cc:492] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:40.571231 6326 ts_tablet_manager.cc:1367] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.009s
I20240503 08:12:40.572284 6326 raft_consensus.cc:348] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:40.572299 6311 raft_consensus.cc:348] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:40.572528 6311 raft_consensus.cc:374] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.572646 6311 raft_consensus.cc:729] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.572836 6326 raft_consensus.cc:374] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:40.572934 6311 consensus_queue.cc:260] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:40.573201 6326 raft_consensus.cc:729] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:40.573704 6311 ts_tablet_manager.cc:1398] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 08:12:40.574359 6326 consensus_queue.cc:260] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:40.581416 6326 ts_tablet_manager.cc:1398] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 08:12:40.613822 5948 raft_consensus.cc:482] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:40.614045 5948 raft_consensus.cc:504] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:40.614816 5948 leader_election.cc:288] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:40.615770 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e46a8bd08d4d5d9b3e5b46db4cda61" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:40.616004 2201 raft_consensus.cc:2454] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:40.616444 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e46a8bd08d4d5d9b3e5b46db4cda61" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:40.616480 4636 leader_election.cc:302] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:40.616662 4965 raft_consensus.cc:2454] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:40.616806 5948 raft_consensus.cc:2781] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:40.616937 5948 raft_consensus.cc:482] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:40.617034 5948 raft_consensus.cc:3037] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.620615 6350 tablet_bootstrap.cc:492] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:40.620450 5948 raft_consensus.cc:504] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:40.621181 6350 tablet_bootstrap.cc:492] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:12:40.621330 5948 leader_election.cc:288] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:40.621490 6350 ts_tablet_manager.cc:1367] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.137s user 0.046s sys 0.034s
I20240503 08:12:40.622378 6350 raft_consensus.cc:348] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:40.622606 6350 raft_consensus.cc:729] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:12:40.622814 6350 consensus_queue.cc:260] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:40.623463 6350 ts_tablet_manager.cc:1398] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240503 08:12:40.624258 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e46a8bd08d4d5d9b3e5b46db4cda61" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:40.624477 4965 raft_consensus.cc:3037] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.629405 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e46a8bd08d4d5d9b3e5b46db4cda61" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:40.629724 2201 raft_consensus.cc:3037] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:40.633369 2201 raft_consensus.cc:2454] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:40.635622 4636 leader_election.cc:302] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:40.636052 2224 tablet_copy_service.cc:342] P 184c543cb05d49cc98ba1c56617be7a5: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-314d235cdca241f8b80a7046fb1f524c received from {username='slave'} at 127.27.157.198:51901
I20240503 08:12:40.636118 5948 raft_consensus.cc:2781] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:40.636232 2224 tablet_copy_service.cc:433] P 184c543cb05d49cc98ba1c56617be7a5: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-314d235cdca241f8b80a7046fb1f524c on tablet 314d235cdca241f8b80a7046fb1f524c with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:40.636651 5948 raft_consensus.cc:686] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:40.637144 5948 consensus_queue.cc:237] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:40.637666 4965 raft_consensus.cc:2454] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:40.644605 1664 catalog_manager.cc:5484] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:40.648594 1942 consensus_queue.cc:237] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:40.651399 4834 raft_consensus.cc:1261] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:12:40.652431 5948 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:40.657416 2201 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:40.657310 1812 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:40.658587 6180 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:40.659358 6214 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:12:40.660939 1878 consensus_peers.cc:485] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cf9b2eef42e4fff8cdf5f9dd7b8b31e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:40.678097 6378 ts_tablet_manager.cc:902] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:40.680945 6180 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:40.681638 1812 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:40.698330 1667 catalog_manager.cc:5484] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:40.701352 2204 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:40.707309 4568 raft_consensus.cc:1261] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab [term 2 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:12:40.708070 5948 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:40.745390 5786 consensus_queue.cc:1035] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:40.752867 1812 raft_consensus.cc:1261] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:12:40.753104 6378 tablet_copy_client.cc:308] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:12:40.756166 1812 consensus_queue.cc:237] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:40.757848 5948 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20240503 08:12:40.766073 1667 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240503 08:12:40.774696 1746 consensus_peers.cc:485] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fc8279b96f74e879b499b0143502c8f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:40.776252 2074 raft_consensus.cc:1261] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:40.777993 4965 raft_consensus.cc:1261] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:40.778678 6096 consensus_queue.cc:1035] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:40.779017 5948 consensus_queue.cc:1035] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:40.783226 5931 consensus_queue.cc:1035] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:40.793686 1942 raft_consensus.cc:1261] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:40.794510 6096 consensus_queue.cc:1035] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:40.798419 6403 ts_tablet_manager.cc:902] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:40.828544 6406 ts_tablet_manager.cc:902] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:40.839910 6348 consensus_queue.cc:1035] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:40.851586 1667 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 (table rebalancer_dynamic_table [id=bb5ca2c4a6124ffaa9fde8ae5a10494a]). Expected version 1 got 0
I20240503 08:12:40.861213 1942 raft_consensus.cc:2932] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:40.844034 6409 raft_consensus.cc:2932] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:40.872558 2201 raft_consensus.cc:1261] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:40.877816 6270 consensus_queue.cc:1035] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:40.894153 5983 consensus_queue.cc:1035] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:40.901381 1664 catalog_manager.cc:5484] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:40.921736 1667 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 (table rebalancer_dynamic_table [id=bb5ca2c4a6124ffaa9fde8ae5a10494a]). Expected version 1 got 0
I20240503 08:12:40.923895 6412 tablet.cc:1722] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:40.924589 1667 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 (table rebalancer_dynamic_table [id=bb5ca2c4a6124ffaa9fde8ae5a10494a]). Expected version 1 got 0
I20240503 08:12:40.924795 6218 consensus_queue.cc:1035] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:40.926052 1747 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.195:43117, user_credentials={real_user=slave}} blocked reactor thread for 80568.2us
I20240503 08:12:40.920060 6413 tablet.cc:1722] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:40.929316 6413 tablet.cc:1620] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:40.935062 6412 tablet.cc:1620] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:40.939589 2074 raft_consensus.cc:2932] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
W20240503 08:12:40.947796 1748 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.194:41975, user_credentials={real_user=slave}} blocked reactor thread for 50021.1us
I20240503 08:12:40.951180 5357 consensus_queue.cc:1035] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:40.963303 6406 tablet_copy_client.cc:308] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:40.964565 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 044d8e5a69444865980ffc06a6be9080 from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:52311)
I20240503 08:12:40.964833 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 044d8e5a69444865980ffc06a6be9080 from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:52311: session id = f24648c92c384f1ba349eee801f985e2-044d8e5a69444865980ffc06a6be9080
I20240503 08:12:40.974421 2093 tablet_copy_source_session.cc:205] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:40.980660 6406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 044d8e5a69444865980ffc06a6be9080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:40.996932 6430 tablet.cc:1722] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:40.997742 6350 ts_tablet_manager.cc:902] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:40.999804 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet 54fe406b5f324d629243f7f939535620 from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:55553)
I20240503 08:12:41.000159 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet 54fe406b5f324d629243f7f939535620 from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:55553: session id = 184c543cb05d49cc98ba1c56617be7a5-54fe406b5f324d629243f7f939535620
I20240503 08:12:41.005456 4855 tablet_copy_source_session.cc:205] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:41.008023 6430 tablet.cc:1620] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: MemRowSet was empty: no flush needed.
I20240503 08:12:41.009203 6378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54fe406b5f324d629243f7f939535620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.013492 6406 tablet_copy_client.cc:783] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:41.014571 6406 tablet_copy_client.cc:647] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:41.020594 6406 tablet_copy_client.cc:523] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:41.023972 1672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:41.026319 6378 tablet_copy_client.cc:783] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:41.026804 6378 tablet_copy_client.cc:647] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:41.028924 4703 raft_consensus.cc:1203] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:41.041260 6350 tablet_copy_client.cc:308] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:41.042272 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 133911d44b394e24a89a5d9443230d32 from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:35383)
I20240503 08:12:41.042563 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 133911d44b394e24a89a5d9443230d32 from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:35383: session id = 31f41542452b4b1eb994ff4223d039fc-133911d44b394e24a89a5d9443230d32
I20240503 08:12:41.044848 6406 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:41.046514 2093 tablet_copy_source_session.cc:205] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:41.047617 1671 catalog_manager.cc:5576] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183) does not have the latest schema for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 (table rebalancer_dynamic_table [id=bb5ca2c4a6124ffaa9fde8ae5a10494a]). Expected version 1 got 0
I20240503 08:12:41.049170 6350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133911d44b394e24a89a5d9443230d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.050711 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 437037e814464cf5959153bcd3efc145 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=f292cdedaf1b46d3a5d119cefab5e3c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.051081 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 437037e814464cf5959153bcd3efc145 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=f292cdedaf1b46d3a5d119cefab5e3c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.051339 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437037e814464cf5959153bcd3efc145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.051645 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437037e814464cf5959153bcd3efc145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.056856 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 34e1c5dc09a54380a17f2cf08a473d57 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=f292cdedaf1b46d3a5d119cefab5e3c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.057410 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e1c5dc09a54380a17f2cf08a473d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.058738 4547 tablet_service.cc:1467] Processing CreateTablet for tablet 34e1c5dc09a54380a17f2cf08a473d57 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=f292cdedaf1b46d3a5d119cefab5e3c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.059342 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e1c5dc09a54380a17f2cf08a473d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.064916 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 34e1c5dc09a54380a17f2cf08a473d57 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=f292cdedaf1b46d3a5d119cefab5e3c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.065744 4549 tablet_service.cc:1467] Processing CreateTablet for tablet 01821d71873d4ad78d7b7f685161a229 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=f292cdedaf1b46d3a5d119cefab5e3c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.066197 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01821d71873d4ad78d7b7f685161a229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.066359 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 01821d71873d4ad78d7b7f685161a229 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=f292cdedaf1b46d3a5d119cefab5e3c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.066936 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01821d71873d4ad78d7b7f685161a229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.082073 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 437037e814464cf5959153bcd3efc145 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=f292cdedaf1b46d3a5d119cefab5e3c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.082696 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437037e814464cf5959153bcd3efc145. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.082978 1664 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=bb5ca2c4a6124ffaa9fde8ae5a10494a] alter complete (version 1)
I20240503 08:12:41.088404 6378 tablet_copy_client.cc:523] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:41.093009 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e1c5dc09a54380a17f2cf08a473d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.093467 6441 tablet_bootstrap.cc:492] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:41.097344 6441 tablet_bootstrap.cc:654] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.097785 4681 tablet_service.cc:1467] Processing CreateTablet for tablet 01821d71873d4ad78d7b7f685161a229 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=f292cdedaf1b46d3a5d119cefab5e3c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:41.098346 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01821d71873d4ad78d7b7f685161a229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.104423 6443 tablet_bootstrap.cc:492] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:41.106977 6443 tablet_bootstrap.cc:654] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.108580 6440 tablet_bootstrap.cc:492] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:41.109206 6378 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:41.111063 6440 tablet_bootstrap.cc:654] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.116418 6445 tablet_bootstrap.cc:492] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:41.119138 6350 tablet_copy_client.cc:783] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:41.121008 6441 tablet_bootstrap.cc:492] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:41.122828 6403 tablet_copy_client.cc:308] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:12:41.128513 6441 ts_tablet_manager.cc:1367] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.006s
I20240503 08:12:41.131223 6439 tablet_bootstrap.cc:492] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:41.132613 6350 tablet_copy_client.cc:647] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:41.134711 6441 raft_consensus.cc:348] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.136070 6350 tablet_copy_client.cc:523] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:41.137213 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet f161c8da3b0e4b12b85ca0420a86dc07 from peer 43e2e01cdd314f9c872c4336ef74250a ({username='slave'} at 127.27.157.195:36783)
I20240503 08:12:41.137777 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet f161c8da3b0e4b12b85ca0420a86dc07 from peer 43e2e01cdd314f9c872c4336ef74250a at {username='slave'} at 127.27.157.195:36783: session id = 43e2e01cdd314f9c872c4336ef74250a-f161c8da3b0e4b12b85ca0420a86dc07
I20240503 08:12:41.138450 6441 raft_consensus.cc:374] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.138633 6441 raft_consensus.cc:729] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.138979 6441 consensus_queue.cc:260] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.139364 6445 tablet_bootstrap.cc:654] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.141304 4855 tablet_copy_source_session.cc:205] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:41.142761 6448 ts_tablet_manager.cc:902] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:41.145952 6403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f161c8da3b0e4b12b85ca0420a86dc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.153263 6438 tablet_bootstrap.cc:492] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:41.155834 6440 tablet_bootstrap.cc:492] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:41.156170 6440 ts_tablet_manager.cc:1367] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.000s
I20240503 08:12:41.157198 6440 raft_consensus.cc:348] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:41.157503 6440 raft_consensus.cc:374] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.157631 6440 raft_consensus.cc:729] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.157967 6440 consensus_queue.cc:260] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:41.160584 6441 ts_tablet_manager.cc:1398] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240503 08:12:41.160643 6445 tablet_bootstrap.cc:492] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:41.162806 6443 tablet_bootstrap.cc:492] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:41.163094 6443 ts_tablet_manager.cc:1367] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.004s
I20240503 08:12:41.164148 6443 raft_consensus.cc:348] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:41.164512 6443 raft_consensus.cc:374] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.164698 6443 raft_consensus.cc:729] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.165082 6443 consensus_queue.cc:260] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:41.165853 6443 ts_tablet_manager.cc:1398] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:41.166074 6403 tablet_copy_client.cc:783] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:41.166260 6440 ts_tablet_manager.cc:1398] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:41.166720 6403 tablet_copy_client.cc:647] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:41.167783 6439 tablet_bootstrap.cc:654] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.162725 6445 ts_tablet_manager.cc:1367] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.007s
I20240503 08:12:41.170248 6403 tablet_copy_client.cc:523] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:41.180385 6378 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:41.180738 6445 raft_consensus.cc:348] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.181731 6378 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:12:41.182142 6445 raft_consensus.cc:374] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.183419 6445 raft_consensus.cc:729] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.184271 6378 ts_tablet_manager.cc:1367] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.007s
I20240503 08:12:41.183857 6445 consensus_queue.cc:260] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.185984 6378 raft_consensus.cc:348] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:41.188779 6350 tablet_bootstrap.cc:492] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:41.190449 6403 tablet_bootstrap.cc:492] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:41.190781 6445 ts_tablet_manager.cc:1398] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240503 08:12:41.191466 6438 tablet_bootstrap.cc:654] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.193521 6445 tablet_bootstrap.cc:492] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:41.194912 6378 raft_consensus.cc:729] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:12:41.198035 6378 consensus_queue.cc:260] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:41.198486 6439 tablet_bootstrap.cc:492] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:41.198769 6439 ts_tablet_manager.cc:1367] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.002s
I20240503 08:12:41.202307 6439 raft_consensus.cc:348] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.205513 6378 ts_tablet_manager.cc:1398] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240503 08:12:41.208267 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-54fe406b5f324d629243f7f939535620 received from {username='slave'} at 127.27.157.196:55553
I20240503 08:12:41.208480 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-54fe406b5f324d629243f7f939535620 on tablet 54fe406b5f324d629243f7f939535620 with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:41.210283 6445 tablet_bootstrap.cc:654] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.213413 1672 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:41.220974 6439 raft_consensus.cc:374] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.221259 6439 raft_consensus.cc:729] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.221673 6439 consensus_queue.cc:260] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.229449 6438 tablet_bootstrap.cc:492] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:41.229790 6438 ts_tablet_manager.cc:1367] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.083s user 0.000s sys 0.006s
I20240503 08:12:41.231165 6438 raft_consensus.cc:348] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.231675 6438 raft_consensus.cc:374] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.231662 1672 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 (table rebalancer_dynamic_table [id=bb5ca2c4a6124ffaa9fde8ae5a10494a]). Expected version 2 got 1
I20240503 08:12:41.231982 6438 raft_consensus.cc:729] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.234026 6412 tablet.cc:1722] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:41.234722 6412 tablet.cc:1620] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:41.242174 6430 tablet.cc:1722] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:41.242797 6430 tablet.cc:1620] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: MemRowSet was empty: no flush needed.
I20240503 08:12:41.244115 6445 tablet_bootstrap.cc:492] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:41.246013 6445 ts_tablet_manager.cc:1367] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.004s
I20240503 08:12:41.247303 6445 raft_consensus.cc:348] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.247738 6445 raft_consensus.cc:374] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.247900 6445 raft_consensus.cc:729] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.248293 6445 consensus_queue.cc:260] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.249411 6445 ts_tablet_manager.cc:1398] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 08:12:41.249414 6413 tablet.cc:1722] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:41.250006 6413 tablet.cc:1620] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:41.250725 6403 tablet_bootstrap.cc:492] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:41.251227 6271 raft_consensus.cc:482] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.251458 6403 tablet_bootstrap.cc:492] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap complete.
I20240503 08:12:41.251511 6271 raft_consensus.cc:504] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.251832 6403 ts_tablet_manager.cc:1367] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.006s
I20240503 08:12:41.252436 6271 leader_election.cc:288] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:41.253126 6403 raft_consensus.cc:348] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:41.253394 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01821d71873d4ad78d7b7f685161a229" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:41.253609 6403 raft_consensus.cc:729] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: LEARNER
I20240503 08:12:41.253731 2074 raft_consensus.cc:2454] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:41.254030 6403 consensus_queue.cc:260] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:41.254631 4636 leader_election.cc:302] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:41.255048 6271 raft_consensus.cc:2781] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:41.255242 6271 raft_consensus.cc:482] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:41.255398 6271 raft_consensus.cc:3037] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.255306 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01821d71873d4ad78d7b7f685161a229" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:41.255717 4568 raft_consensus.cc:2454] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:41.257860 6403 ts_tablet_manager.cc:1398] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:41.256666 2204 consensus_queue.cc:237] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:41.259107 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 43e2e01cdd314f9c872c4336ef74250a-f161c8da3b0e4b12b85ca0420a86dc07 received from {username='slave'} at 127.27.157.195:36783
I20240503 08:12:41.259270 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 43e2e01cdd314f9c872c4336ef74250a-f161c8da3b0e4b12b85ca0420a86dc07 on tablet f161c8da3b0e4b12b85ca0420a86dc07 with peer 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:12:41.261672 2074 raft_consensus.cc:1203] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:41.270181 6428 raft_consensus.cc:482] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.270392 6428 raft_consensus.cc:504] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:41.271309 6428 leader_election.cc:288] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:41.274241 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437037e814464cf5959153bcd3efc145" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:41.274505 1813 raft_consensus.cc:2454] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:41.274379 6271 raft_consensus.cc:504] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.274758 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=bb5ca2c4a6124ffaa9fde8ae5a10494a] alter complete (version 2)
I20240503 08:12:41.275104 4902 leader_election.cc:302] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:41.275573 6348 raft_consensus.cc:2781] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:41.275699 6348 raft_consensus.cc:482] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:41.275800 6348 raft_consensus.cc:3037] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.276970 6438 consensus_queue.cc:260] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.277365 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01821d71873d4ad78d7b7f685161a229" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:41.277565 2074 raft_consensus.cc:3037] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.280606 2074 raft_consensus.cc:2454] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:41.280640 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437037e814464cf5959153bcd3efc145" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
W20240503 08:12:41.281263 4900 leader_election.cc:341] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:41.281278 4636 leader_election.cc:302] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:41.281793 6270 raft_consensus.cc:2781] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:41.282009 6270 raft_consensus.cc:686] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:41.282230 6348 raft_consensus.cc:504] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:41.282418 6270 consensus_queue.cc:237] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.282655 6438 ts_tablet_manager.cc:1398] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20240503 08:12:41.282949 6348 leader_election.cc:288] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:41.283504 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437037e814464cf5959153bcd3efc145" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
W20240503 08:12:41.284602 4900 leader_election.cc:341] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:41.284638 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437037e814464cf5959153bcd3efc145" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:41.284879 1813 raft_consensus.cc:3037] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.286197 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01821d71873d4ad78d7b7f685161a229" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:41.286517 4569 raft_consensus.cc:3037] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.289038 1813 raft_consensus.cc:2454] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:41.289667 4902 leader_election.cc:302] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, b8f581390da449ccb9e299011296405e; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:41.292085 4569 raft_consensus.cc:2454] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:41.294291 6439 ts_tablet_manager.cc:1398] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.095s user 0.002s sys 0.001s
I20240503 08:12:41.294355 6428 raft_consensus.cc:2781] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:41.294574 6271 leader_election.cc:288] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:41.295536 6428 raft_consensus.cc:686] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:41.296005 6428 consensus_queue.cc:237] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:41.297968 6439 tablet_bootstrap.cc:492] T 34e1c5dc09a54380a17f2cf08a473d57 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
W20240503 08:12:41.303272 2140 consensus_peers.cc:485] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 913a3aacff964ac9a321fac187df33ee. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:41.303717 1813 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:41.303774 1942 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:41.304677 6301 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:41.305558 6301 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:41.330045 1664 catalog_manager.cc:5484] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:41.338848 6422 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:41.339495 1813 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:41.348346 1671 catalog_manager.cc:5484] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:41.349802 1942 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:41.363559 1664 catalog_manager.cc:5484] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:41.365857 6466 ts_tablet_manager.cc:902] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:41.367520 6448 tablet_copy_client.cc:308] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:41.368511 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:43591)
I20240503 08:12:41.368841 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:43591: session id = 00b61625d8a7460892c118a2570afda5-1cf9b2eef42e4fff8cdf5f9dd7b8b31e
I20240503 08:12:41.369148 6439 tablet_bootstrap.cc:654] T 34e1c5dc09a54380a17f2cf08a473d57 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.374846 1964 tablet_copy_source_session.cc:205] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:41.379463 6448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.391721 1671 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:41.404229 6412 tablet.cc:1722] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:41.406785 6448 tablet_copy_client.cc:783] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:41.407673 6448 tablet_copy_client.cc:647] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:41.410153 6448 tablet_copy_client.cc:523] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:41.411579 6413 tablet.cc:1722] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:41.412102 6439 tablet_bootstrap.cc:492] T 34e1c5dc09a54380a17f2cf08a473d57 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:41.412348 6439 ts_tablet_manager.cc:1367] T 34e1c5dc09a54380a17f2cf08a473d57 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.114s user 0.005s sys 0.001s
I20240503 08:12:41.412717 6430 tablet.cc:1722] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:41.413398 6439 raft_consensus.cc:348] T 34e1c5dc09a54380a17f2cf08a473d57 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.414395 6439 raft_consensus.cc:374] T 34e1c5dc09a54380a17f2cf08a473d57 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.414597 6439 raft_consensus.cc:729] T 34e1c5dc09a54380a17f2cf08a473d57 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.415084 6439 consensus_queue.cc:260] T 34e1c5dc09a54380a17f2cf08a473d57 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.416821 6448 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:41.417122 6439 ts_tablet_manager.cc:1398] T 34e1c5dc09a54380a17f2cf08a473d57 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:41.417584 6439 tablet_bootstrap.cc:492] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:41.420553 6439 tablet_bootstrap.cc:654] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.431793 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=bb5ca2c4a6124ffaa9fde8ae5a10494a] alter complete (version 3)
I20240503 08:12:41.435513 6439 tablet_bootstrap.cc:492] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:41.436393 6439 ts_tablet_manager.cc:1367] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.006s
I20240503 08:12:41.436174 6466 tablet_copy_client.cc:308] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:41181
I20240503 08:12:41.437714 1833 tablet_copy_service.cc:140] P b8f581390da449ccb9e299011296405e: Received BeginTabletCopySession request for tablet 8fc8279b96f74e879b499b0143502c8f from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:43085)
I20240503 08:12:41.437901 1833 tablet_copy_service.cc:161] P b8f581390da449ccb9e299011296405e: Beginning new tablet copy session on tablet 8fc8279b96f74e879b499b0143502c8f from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:43085: session id = 5f95b05d111e4090bbc59dec394ed0ab-8fc8279b96f74e879b499b0143502c8f
I20240503 08:12:41.437839 6439 raft_consensus.cc:348] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:41.438201 6439 raft_consensus.cc:374] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.438302 6439 raft_consensus.cc:729] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.438587 6439 consensus_queue.cc:260] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:41.441207 1833 tablet_copy_source_session.cc:205] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:41.441301 6439 ts_tablet_manager.cc:1398] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:41.441855 4703 consensus_queue.cc:237] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:41.444649 6466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc8279b96f74e879b499b0143502c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.445053 4568 raft_consensus.cc:1261] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab [term 2 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:12:41.445968 6469 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:41.447685 1813 raft_consensus.cc:1261] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:12:41.448807 6469 consensus_queue.cc:1035] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:41.452584 6469 raft_consensus.cc:2932] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:41.454947 6466 tablet_copy_client.cc:783] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:41.455247 6466 tablet_copy_client.cc:647] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:41.458600 4568 raft_consensus.cc:2932] T f6cff57835b846e6b48c08bef52f5b68 P 5f95b05d111e4090bbc59dec394ed0ab [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:41.460459 1812 raft_consensus.cc:2932] T f6cff57835b846e6b48c08bef52f5b68 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:41.468703 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f6cff57835b846e6b48c08bef52f5b68 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:41.469868 1671 catalog_manager.cc:5484] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 3 to 5, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New cstate: current_term: 2 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:41.470638 6466 tablet_copy_client.cc:523] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:41.475229 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cff57835b846e6b48c08bef52f5b68 with delete_type TABLET_DATA_TOMBSTONED (TS f24648c92c384f1ba349eee801f985e2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:41.476332 6466 tablet_bootstrap.cc:492] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:41.489674 6487 tablet_replica.cc:330] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:41.492089 6469 raft_consensus.cc:482] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.492192 6487 raft_consensus.cc:2229] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.492336 6469 raft_consensus.cc:504] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.492995 6487 raft_consensus.cc:2258] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.493664 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e1c5dc09a54380a17f2cf08a473d57" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:41.493922 4835 raft_consensus.cc:2454] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:41.494459 4638 leader_election.cc:302] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:41.494864 6271 raft_consensus.cc:2781] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:41.494959 6271 raft_consensus.cc:482] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:41.495060 6271 raft_consensus.cc:3037] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.498169 6271 raft_consensus.cc:504] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.499619 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e1c5dc09a54380a17f2cf08a473d57" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:41.499915 4835 raft_consensus.cc:3037] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.505668 6271 leader_election.cc:288] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:41.505990 4835 raft_consensus.cc:2454] T 34e1c5dc09a54380a17f2cf08a473d57 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:41.506604 4638 leader_election.cc:302] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:41.507026 6271 raft_consensus.cc:2781] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:41.507934 6469 leader_election.cc:288] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:41.508038 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e1c5dc09a54380a17f2cf08a473d57" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:41.508144 6271 raft_consensus.cc:686] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:41.508344 4568 raft_consensus.cc:3037] T 34e1c5dc09a54380a17f2cf08a473d57 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.508452 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e1c5dc09a54380a17f2cf08a473d57" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:41.508702 6271 consensus_queue.cc:237] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:41.511860 4568 raft_consensus.cc:2454] T 34e1c5dc09a54380a17f2cf08a473d57 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:41.519788 6487 ts_tablet_manager.cc:1875] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:41.527755 1672 catalog_manager.cc:5484] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:41.545773 6448 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:41.546623 6448 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:12:41.547191 6448 ts_tablet_manager.cc:1367] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.130s user 0.053s sys 0.007s
I20240503 08:12:41.548511 6448 raft_consensus.cc:348] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:41.549038 6448 raft_consensus.cc:729] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:12:41.549388 6448 consensus_queue.cc:260] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:41.552090 1664 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:41.552968 6448 ts_tablet_manager.cc:1398] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:41.552379 1664 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:41.554255 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-1cf9b2eef42e4fff8cdf5f9dd7b8b31e received from {username='slave'} at 127.27.157.200:43591
I20240503 08:12:41.554428 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 00b61625d8a7460892c118a2570afda5-1cf9b2eef42e4fff8cdf5f9dd7b8b31e on tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:12:41.568084 6422 raft_consensus.cc:1050] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:12:41.569494 6422 consensus_queue.cc:237] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:41.577245 1664 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 28e46a8bd08d4d5d9b3e5b46db4cda61
I20240503 08:12:41.578183 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:41.578598 4946 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:41.579607 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:41 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:41.598723 2073 consensus_queue.cc:237] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:41.601613 6493 tablet_replica.cc:330] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:41.601897 4703 raft_consensus.cc:1261] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:41.605629 4703 raft_consensus.cc:1261] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:41.607596 6491 tablet_replica.cc:330] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:41.607632 5357 consensus_queue.cc:1035] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:41.607774 6493 raft_consensus.cc:2229] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.607805 6491 raft_consensus.cc:2229] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:41.602939 6431 consensus_queue.cc:1035] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:41.608528 6493 raft_consensus.cc:2258] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.609673 1944 raft_consensus.cc:1261] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:41.612506 6492 tablet_replica.cc:330] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:41.612763 6492 raft_consensus.cc:2229] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.613004 6492 raft_consensus.cc:2258] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.613663 6491 raft_consensus.cc:2258] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.614235 1812 raft_consensus.cc:1261] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:12:41.614897 2007 consensus_peers.cc:485] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79579b65538d47f7973fa51f973b3820. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:41.615433 6491 ts_tablet_manager.cc:1875] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.618355 5874 consensus_queue.cc:1035] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:41.621228 2073 raft_consensus.cc:1261] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:41.635454 6487 ts_tablet_manager.cc:1888] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 08:12:41.635768 6487 log.cc:1196] T f6cff57835b846e6b48c08bef52f5b68 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/f6cff57835b846e6b48c08bef52f5b68
I20240503 08:12:41.637017 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet f6cff57835b846e6b48c08bef52f5b68 (table rebalancer_extra_table_6 [id=6b5d1749a8944d598384a9bb153c5fc1]) successfully deleted
I20240503 08:12:41.638242 6431 consensus_queue.cc:1035] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20240503 08:12:41.641330 6301 consensus_queue.cc:1035] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:41.644517 4965 raft_consensus.cc:1203] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:41.648442 6493 ts_tablet_manager.cc:1875] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.657644 6492 ts_tablet_manager.cc:1875] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:41.677150 5357 raft_consensus.cc:2932] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:41.680011 1664 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:41.680131 1813 raft_consensus.cc:2932] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:41.688148 6491 ts_tablet_manager.cc:1888] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:41.691789 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7ed763c33a429682e5c6a44668cd48 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30]), partition=
I20240503 08:12:41.692202 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7ed763c33a429682e5c6a44668cd48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.692631 2204 raft_consensus.cc:1203] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:41.693773 4549 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7ed763c33a429682e5c6a44668cd48 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30]), partition=
I20240503 08:12:41.694147 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7ed763c33a429682e5c6a44668cd48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.697696 1671 catalog_manager.cc:5484] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:41.705173 6439 tablet_bootstrap.cc:492] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:41.706976 6439 tablet_bootstrap.cc:654] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.714547 6492 ts_tablet_manager.cc:1888] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:41.714736 6492 log.cc:1196] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/28e46a8bd08d4d5d9b3e5b46db4cda61
I20240503 08:12:41.715237 6492 ts_tablet_manager.cc:1909] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:41.715570 6493 ts_tablet_manager.cc:1888] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:41.715754 6493 log.cc:1196] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/28e46a8bd08d4d5d9b3e5b46db4cda61
I20240503 08:12:41.718629 6493 ts_tablet_manager.cc:1909] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:41.691265 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7ed763c33a429682e5c6a44668cd48 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30]), partition=
I20240503 08:12:41.720716 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7ed763c33a429682e5c6a44668cd48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.715957 6350 tablet_bootstrap.cc:492] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:41.721375 6350 tablet_bootstrap.cc:492] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:12:41.721370 6439 tablet_bootstrap.cc:492] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:41.721575 6439 ts_tablet_manager.cc:1367] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.007s
I20240503 08:12:41.721689 6350 ts_tablet_manager.cc:1367] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.533s user 0.053s sys 0.003s
I20240503 08:12:41.722563 6350 raft_consensus.cc:348] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:41.722815 6350 raft_consensus.cc:729] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:12:41.722518 6439 raft_consensus.cc:348] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:41.723080 6439 raft_consensus.cc:374] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.723162 6439 raft_consensus.cc:729] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.723531 6439 consensus_queue.cc:260] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:41.728684 6350 consensus_queue.cc:260] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:41.731719 6350 ts_tablet_manager.cc:1398] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 08:12:41.732805 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-133911d44b394e24a89a5d9443230d32 received from {username='slave'} at 127.27.157.198:35383
I20240503 08:12:41.732947 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-133911d44b394e24a89a5d9443230d32 on tablet 133911d44b394e24a89a5d9443230d32 with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:41.734607 6445 tablet_bootstrap.cc:492] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:41.734763 6491 log.cc:1196] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/28e46a8bd08d4d5d9b3e5b46db4cda61
I20240503 08:12:41.735294 6491 ts_tablet_manager.cc:1909] T 28e46a8bd08d4d5d9b3e5b46db4cda61 P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:41.736487 6445 tablet_bootstrap.cc:654] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.738662 6439 ts_tablet_manager.cc:1398] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.017s user 0.001s sys 0.002s
I20240503 08:12:41.740267 6495 raft_consensus.cc:2932] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:41.741075 1942 raft_consensus.cc:2932] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:41.745023 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 (table rebalancer_dynamic_table_new_name [id=bb5ca2c4a6124ffaa9fde8ae5a10494a]) successfully deleted
I20240503 08:12:41.745684 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 (table rebalancer_dynamic_table_new_name [id=bb5ca2c4a6124ffaa9fde8ae5a10494a]) successfully deleted
I20240503 08:12:41.749814 2073 raft_consensus.cc:2932] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:41.754276 1671 catalog_manager.cc:5484] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:41.754704 6406 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:41.755435 6406 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:12:41.755776 6406 ts_tablet_manager.cc:1367] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.711s user 0.061s sys 0.004s
I20240503 08:12:41.757872 6406 raft_consensus.cc:348] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:41.758203 6406 raft_consensus.cc:729] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:12:41.758437 6406 consensus_queue.cc:260] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:41.759058 6406 ts_tablet_manager.cc:1398] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:41.760985 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-044d8e5a69444865980ffc06a6be9080 received from {username='slave'} at 127.27.157.199:52311
I20240503 08:12:41.761163 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session f24648c92c384f1ba349eee801f985e2-044d8e5a69444865980ffc06a6be9080 on tablet 044d8e5a69444865980ffc06a6be9080 with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:12:41.761466 6094 raft_consensus.cc:482] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.761658 6094 raft_consensus.cc:504] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:41.762454 6094 leader_election.cc:288] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:41.763988 6019 raft_consensus.cc:1050] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 43e2e01cdd314f9c872c4336ef74250a to VOTER
I20240503 08:12:41.764963 6019 consensus_queue.cc:237] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:12:41.771250 6466 tablet_bootstrap.cc:492] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:41.771852 6466 tablet_bootstrap.cc:492] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:41.772198 6466 ts_tablet_manager.cc:1367] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.296s user 0.092s sys 0.015s
I20240503 08:12:41.773169 6466 raft_consensus.cc:348] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:41.773455 6466 raft_consensus.cc:729] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:41.773701 6466 consensus_queue.cc:260] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:41.774303 6466 ts_tablet_manager.cc:1398] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:41.777571 1833 tablet_copy_service.cc:342] P b8f581390da449ccb9e299011296405e: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-8fc8279b96f74e879b499b0143502c8f received from {username='slave'} at 127.27.157.197:43085
I20240503 08:12:41.777781 1833 tablet_copy_service.cc:433] P b8f581390da449ccb9e299011296405e: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-8fc8279b96f74e879b499b0143502c8f on tablet 8fc8279b96f74e879b499b0143502c8f with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:41.788252 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437037e814464cf5959153bcd3efc145" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:41.788661 1812 raft_consensus.cc:2379] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f95b05d111e4090bbc59dec394ed0ab in current term 1: Already voted for candidate 00b61625d8a7460892c118a2570afda5 in this term.
I20240503 08:12:41.792572 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437037e814464cf5959153bcd3efc145" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:41.793853 4502 leader_election.cc:302] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab; no voters: 00b61625d8a7460892c118a2570afda5, b8f581390da449ccb9e299011296405e
I20240503 08:12:41.795011 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 28e46a8bd08d4d5d9b3e5b46db4cda61 (table rebalancer_dynamic_table_new_name [id=bb5ca2c4a6124ffaa9fde8ae5a10494a]) successfully deleted
I20240503 08:12:41.799429 6094 raft_consensus.cc:3037] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.803158 6094 raft_consensus.cc:2726] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:41.803699 4703 raft_consensus.cc:2932] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:41.807730 2204 consensus_queue.cc:237] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:41.808498 6501 tablet_bootstrap.cc:492] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:41.810338 6501 tablet_bootstrap.cc:654] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:41.811687 6506 raft_consensus.cc:482] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:41.812868 6506 raft_consensus.cc:504] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:41.813033 4700 raft_consensus.cc:1261] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:41.814075 6495 consensus_queue.cc:1035] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:41.814432 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7ed763c33a429682e5c6a44668cd48" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
W20240503 08:12:41.815629 4502 leader_election.cc:341] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:41.816205 6506 leader_election.cc:288] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:41.816238 2074 raft_consensus.cc:1261] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:41.816373 6445 tablet_bootstrap.cc:492] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:41.816519 6445 ts_tablet_manager.cc:1367] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.007s
I20240503 08:12:41.817167 5931 consensus_queue.cc:1035] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:41.817986 6445 raft_consensus.cc:348] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:41.818234 6445 raft_consensus.cc:374] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.818341 6445 raft_consensus.cc:729] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.818600 6445 consensus_queue.cc:260] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:41.819197 6445 ts_tablet_manager.cc:1398] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:12:41.819563 6495 raft_consensus.cc:2932] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:41.823127 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7ed763c33a429682e5c6a44668cd48" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:41.823551 4702 raft_consensus.cc:2932] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:41.825414 6501 tablet_bootstrap.cc:492] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:41.825579 6501 ts_tablet_manager.cc:1367] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20240503 08:12:41.826479 6501 raft_consensus.cc:348] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:41.826722 6501 raft_consensus.cc:374] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:41.826804 6501 raft_consensus.cc:729] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:41.827077 6501 consensus_queue.cc:260] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:41.829033 6501 ts_tablet_manager.cc:1398] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:12:41.829924 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 314d235cdca241f8b80a7046fb1f524c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:41.832906 1665 catalog_manager.cc:5484] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:41.837491 6406 ts_tablet_manager.cc:902] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:41.840655 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 314d235cdca241f8b80a7046fb1f524c with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:41.845150 1944 raft_consensus.cc:1261] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:41.847731 6431 consensus_queue.cc:1035] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:41.853542 1944 raft_consensus.cc:2932] T 314d235cdca241f8b80a7046fb1f524c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:41.864073 5983 consensus_queue.cc:1035] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:41.865363 4701 raft_consensus.cc:2454] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:41.868379 6512 tablet_replica.cc:330] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:41.868360 4504 leader_election.cc:302] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters: 00b61625d8a7460892c118a2570afda5
I20240503 08:12:41.868629 6512 raft_consensus.cc:2229] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.868891 6512 raft_consensus.cc:2258] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.870106 6512 ts_tablet_manager.cc:1875] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:41.871021 1944 raft_consensus.cc:1261] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:41.871733 5786 consensus_queue.cc:1035] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:41.874867 4567 raft_consensus.cc:1261] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:41.878381 6512 ts_tablet_manager.cc:1888] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:41.878681 6512 log.cc:1196] T 314d235cdca241f8b80a7046fb1f524c P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/314d235cdca241f8b80a7046fb1f524c
I20240503 08:12:41.879505 5931 consensus_queue.cc:1035] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:41.879683 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 314d235cdca241f8b80a7046fb1f524c (table rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]) successfully deleted
I20240503 08:12:41.879889 1943 consensus_queue.cc:237] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:41.884150 6094 raft_consensus.cc:2781] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:41.884685 6094 raft_consensus.cc:482] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:41.884822 6094 raft_consensus.cc:3037] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.885824 5786 raft_consensus.cc:2932] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:41.892158 6094 raft_consensus.cc:504] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:41.892982 6094 leader_election.cc:288] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:41.894304 1944 raft_consensus.cc:2932] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:41.894594 1672 catalog_manager.cc:5484] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:41.895642 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7ed763c33a429682e5c6a44668cd48" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:41.895926 4967 raft_consensus.cc:3037] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.902702 4835 consensus_queue.cc:237] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:12:41.905066 2073 raft_consensus.cc:2932] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:41.916112 6348 consensus_queue.cc:1035] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:41.918643 1944 raft_consensus.cc:1261] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:41.920148 2201 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:41.922405 2074 raft_consensus.cc:1261] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:41.922933 1878 consensus_peers.cc:485] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53197deeb8d7439c90b019a6a90857e0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:41.923054 6406 tablet_copy_client.cc:308] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.196:42183
I20240503 08:12:41.923565 6375 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:41.924571 5931 consensus_queue.cc:1035] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:41.925848 2224 tablet_copy_service.cc:140] P 184c543cb05d49cc98ba1c56617be7a5: Received BeginTabletCopySession request for tablet 913a3aacff964ac9a321fac187df33ee from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:47127)
I20240503 08:12:41.926008 2224 tablet_copy_service.cc:161] P 184c543cb05d49cc98ba1c56617be7a5: Beginning new tablet copy session on tablet 913a3aacff964ac9a321fac187df33ee from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:47127: session id = f24648c92c384f1ba349eee801f985e2-913a3aacff964ac9a321fac187df33ee
I20240503 08:12:41.925884 5931 consensus_queue.cc:1035] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:41.928325 4967 raft_consensus.cc:2454] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:41.929363 4502 leader_election.cc:302] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:41.929991 2224 tablet_copy_source_session.cc:205] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:41.929589 6019 raft_consensus.cc:2932] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:41.930328 6094 raft_consensus.cc:2781] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:41.930501 6094 raft_consensus.cc:686] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:41.930810 6094 consensus_queue.cc:237] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:41.935212 2074 raft_consensus.cc:2932] T f161c8da3b0e4b12b85ca0420a86dc07 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:41.936707 6271 consensus_queue.cc:1035] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.067s
I20240503 08:12:41.942929 1812 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:41.944113 6375 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:41.946274 4835 raft_consensus.cc:1203] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:41.953023 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f161c8da3b0e4b12b85ca0420a86dc07 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:41.955312 1671 catalog_manager.cc:5484] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:41.956825 6406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913a3aacff964ac9a321fac187df33ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:41.962283 1944 raft_consensus.cc:2932] T f161c8da3b0e4b12b85ca0420a86dc07 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:41.962985 6469 consensus_queue.cc:1035] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20240503 08:12:41.966557 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7ed763c33a429682e5c6a44668cd48" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:41.966840 4701 raft_consensus.cc:3037] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:41.966002 5948 consensus_queue.cc:1035] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.043s
I20240503 08:12:41.970901 4699 raft_consensus.cc:2932] T 314d235cdca241f8b80a7046fb1f524c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:41.975858 6406 tablet_copy_client.cc:783] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:41.975859 4549 tablet_service.cc:1514] Processing DeleteTablet for tablet f161c8da3b0e4b12b85ca0420a86dc07 with delete_type TABLET_DATA_TOMBSTONED (TS 5f95b05d111e4090bbc59dec394ed0ab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:41.976155 6406 tablet_copy_client.cc:647] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:41.982177 4703 raft_consensus.cc:1203] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:41.984359 6533 tablet_replica.cc:330] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:41.984768 6533 raft_consensus.cc:2229] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:41.985232 6533 pending_rounds.cc:62] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab: Trying to abort 1 pending ops.
I20240503 08:12:41.985424 6533 pending_rounds.cc:68] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023509347385516032 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f161c8da3b0e4b12b85ca0420a86dc07" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } } }
I20240503 08:12:41.985890 4701 raft_consensus.cc:2454] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:41.986162 6533 raft_consensus.cc:2894] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 08:12:41.986382 6533 raft_consensus.cc:2866] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:12:41.986519 6533 raft_consensus.cc:2258] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:41.988922 6522 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:41.990981 1813 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:42.003598 6406 tablet_copy_client.cc:523] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:42.006341 1672 catalog_manager.cc:5484] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:42.015120 1664 catalog_manager.cc:5484] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:42.016336 6533 ts_tablet_manager.cc:1875] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:42.016788 6406 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:42.033115 2201 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:42.043009 6214 raft_consensus.cc:1050] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:12:42.044077 6214 consensus_queue.cc:237] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:42.046968 6533 ts_tablet_manager.cc:1888] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:42.047160 6533 log.cc:1196] T f161c8da3b0e4b12b85ca0420a86dc07 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/f161c8da3b0e4b12b85ca0420a86dc07
I20240503 08:12:42.055230 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:42.067207 4967 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.068349 5948 consensus_queue.cc:1035] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.068521 6375 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:42.070683 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 415a75c852814ea89b65d294d682c571 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.070997 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 415a75c852814ea89b65d294d682c571 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.071166 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415a75c852814ea89b65d294d682c571. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.071488 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415a75c852814ea89b65d294d682c571. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.083285 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 415a75c852814ea89b65d294d682c571 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.083889 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415a75c852814ea89b65d294d682c571. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.088297 4549 tablet_service.cc:1467] Processing CreateTablet for tablet df06752e8d7247dd962a376c629473c3 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.088515 2204 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.088806 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df06752e8d7247dd962a376c629473c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.089159 6375 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:42.091850 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 84831077defe434c9f29eecfae7b44c8 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.092298 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84831077defe434c9f29eecfae7b44c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.095602 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet f161c8da3b0e4b12b85ca0420a86dc07 (table rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]) successfully deleted
I20240503 08:12:42.097694 2183 tablet_service.cc:1467] Processing CreateTablet for tablet df06752e8d7247dd962a376c629473c3 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.098215 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df06752e8d7247dd962a376c629473c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.098683 4947 tablet_service.cc:1467] Processing CreateTablet for tablet df06752e8d7247dd962a376c629473c3 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.099191 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df06752e8d7247dd962a376c629473c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.104259 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 84831077defe434c9f29eecfae7b44c8 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.104786 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84831077defe434c9f29eecfae7b44c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.105229 1812 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.105911 6214 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:42.107839 4813 tablet_service.cc:1467] Processing CreateTablet for tablet 84831077defe434c9f29eecfae7b44c8 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:42.108341 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84831077defe434c9f29eecfae7b44c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.109334 6501 tablet_bootstrap.cc:492] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:42.111407 6501 tablet_bootstrap.cc:654] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.114389 6501 tablet_bootstrap.cc:492] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:42.114735 6501 ts_tablet_manager.cc:1367] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20240503 08:12:42.115924 6501 raft_consensus.cc:348] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.116377 6501 raft_consensus.cc:374] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.116523 6501 raft_consensus.cc:729] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.116969 6501 consensus_queue.cc:260] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.122193 6501 ts_tablet_manager.cc:1398] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:42.122265 6375 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:42.126634 6501 tablet_bootstrap.cc:492] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:42.128453 6501 tablet_bootstrap.cc:654] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.130831 6439 tablet_bootstrap.cc:492] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:42.133534 6439 tablet_bootstrap.cc:654] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.140868 6556 tablet_bootstrap.cc:492] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:42.140954 1664 catalog_manager.cc:5484] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:42.142050 4965 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:42.143923 4567 raft_consensus.cc:1203] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:42.158049 1812 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:42.160043 2201 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:42.166034 6556 tablet_bootstrap.cc:654] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.166661 6466 ts_tablet_manager.cc:902] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:42.171921 6552 tablet_bootstrap.cc:492] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:42.177119 6551 tablet_bootstrap.cc:492] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:42.185084 6501 tablet_bootstrap.cc:492] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:42.185441 6501 ts_tablet_manager.cc:1367] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.005s
I20240503 08:12:42.186372 6501 raft_consensus.cc:348] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.186674 6501 raft_consensus.cc:374] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.186826 6501 raft_consensus.cc:729] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.187168 6501 consensus_queue.cc:260] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.188113 6439 tablet_bootstrap.cc:492] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:42.188918 6501 ts_tablet_manager.cc:1398] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:42.190815 6501 tablet_bootstrap.cc:492] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:42.197320 6501 tablet_bootstrap.cc:654] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.197938 6439 ts_tablet_manager.cc:1367] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.006s
I20240503 08:12:42.202497 1664 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:42.205897 6551 tablet_bootstrap.cc:654] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.207051 6439 raft_consensus.cc:348] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.207309 6439 raft_consensus.cc:374] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.207448 6439 raft_consensus.cc:729] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.207991 6552 tablet_bootstrap.cc:654] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.207698 6439 consensus_queue.cc:260] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.211098 6439 ts_tablet_manager.cc:1398] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240503 08:12:42.212255 6439 tablet_bootstrap.cc:492] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:42.214929 6556 tablet_bootstrap.cc:492] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:42.215201 6556 ts_tablet_manager.cc:1367] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.074s user 0.006s sys 0.000s
I20240503 08:12:42.216291 6556 raft_consensus.cc:348] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.216682 6556 raft_consensus.cc:374] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.216888 6556 raft_consensus.cc:729] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.217269 6556 consensus_queue.cc:260] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.217434 4701 raft_consensus.cc:1261] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:42.219750 6556 ts_tablet_manager.cc:1398] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:42.223843 4967 raft_consensus.cc:1261] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:42.218143 6094 consensus_queue.cc:1035] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.227474 6506 consensus_queue.cc:1035] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.233850 6501 tablet_bootstrap.cc:492] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:42.234766 6501 ts_tablet_manager.cc:1367] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.002s
I20240503 08:12:42.238131 6501 raft_consensus.cc:348] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.238538 6501 raft_consensus.cc:374] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.238729 6501 raft_consensus.cc:729] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.239146 6501 consensus_queue.cc:260] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.250128 1944 consensus_queue.cc:237] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:42.250751 6501 ts_tablet_manager.cc:1398] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240503 08:12:42.253111 4965 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:42.255728 1813 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:42.256742 6522 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:42.258879 6522 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:42.263557 6552 tablet_bootstrap.cc:492] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:42.263960 6552 ts_tablet_manager.cc:1367] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.092s user 0.006s sys 0.000s
I20240503 08:12:42.265522 6375 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:42.266304 1813 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:42.272606 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:42.276640 1668 catalog_manager.cc:5484] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:42.276772 4965 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:42.282322 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:42.285961 6575 tablet_replica.cc:330] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:42.286343 6575 raft_consensus.cc:2229] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.286760 6575 raft_consensus.cc:2258] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.289342 6439 tablet_bootstrap.cc:654] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.289942 6575 ts_tablet_manager.cc:1875] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:42.297793 1672 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 1f7ed763c33a429682e5c6a44668cd48 (table rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30]). Expected version 1 got 0
I20240503 08:12:42.301008 6019 raft_consensus.cc:1050] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:12:42.302024 6019 consensus_queue.cc:237] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:42.320377 6551 tablet_bootstrap.cc:492] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:42.320822 6551 ts_tablet_manager.cc:1367] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.144s user 0.006s sys 0.000s
I20240503 08:12:42.327114 1672 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 1f7ed763c33a429682e5c6a44668cd48 (table rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30]). Expected version 1 got 0
I20240503 08:12:42.331972 6575 ts_tablet_manager.cc:1888] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:42.333989 6575 log.cc:1196] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/1cf9b2eef42e4fff8cdf5f9dd7b8b31e
I20240503 08:12:42.334334 6552 raft_consensus.cc:348] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.334802 6552 raft_consensus.cc:374] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.335341 6552 raft_consensus.cc:729] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.336197 6552 consensus_queue.cc:260] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.338428 6552 ts_tablet_manager.cc:1398] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.074s user 0.000s sys 0.002s
I20240503 08:12:42.338732 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e (table rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]) successfully deleted
I20240503 08:12:42.349740 6439 tablet_bootstrap.cc:492] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:42.350040 6439 ts_tablet_manager.cc:1367] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.138s user 0.004s sys 0.000s
I20240503 08:12:42.350876 6439 raft_consensus.cc:348] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.351210 6439 raft_consensus.cc:374] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.351346 6439 raft_consensus.cc:729] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.351825 6439 consensus_queue.cc:260] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.355429 6439 ts_tablet_manager.cc:1398] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:42.355654 6551 raft_consensus.cc:348] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.356762 6581 tablet.cc:1722] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:42.359483 6551 raft_consensus.cc:374] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.362834 6578 tablet.cc:1722] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:42.362964 6551 raft_consensus.cc:729] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.363782 6578 tablet.cc:1620] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:42.366210 6551 consensus_queue.cc:260] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.368355 1672 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 1f7ed763c33a429682e5c6a44668cd48 (table rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30]). Expected version 1 got 0
I20240503 08:12:42.373961 1672 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet 1f7ed763c33a429682e5c6a44668cd48 (table rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30]). Expected version 1 got 0
I20240503 08:12:42.376473 1668 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 1f7ed763c33a429682e5c6a44668cd48 (table rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30]). Expected version 1 got 0
I20240503 08:12:42.377049 6551 ts_tablet_manager.cc:1398] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.056s user 0.003s sys 0.000s
I20240503 08:12:42.378597 6551 tablet_bootstrap.cc:492] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:42.388916 1668 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet 1f7ed763c33a429682e5c6a44668cd48 (table rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30]). Expected version 1 got 0
I20240503 08:12:42.393793 6551 tablet_bootstrap.cc:654] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:42.394579 6581 tablet.cc:1620] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:42.395447 2201 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:42.395517 2074 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:42.396225 5786 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:42.404887 6428 raft_consensus.cc:482] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.405073 6428 raft_consensus.cc:504] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.405207 5931 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:42.405844 6428 leader_election.cc:288] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:42.406931 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415a75c852814ea89b65d294d682c571" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:42.407156 4835 raft_consensus.cc:2454] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:42.407814 4965 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:42.408901 4903 leader_election.cc:302] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:42.409713 6588 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:42.409925 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415a75c852814ea89b65d294d682c571" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:42.410154 1944 raft_consensus.cc:2454] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:42.411383 6428 raft_consensus.cc:2781] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.411569 6428 raft_consensus.cc:482] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.411687 6428 raft_consensus.cc:3037] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.418359 6428 raft_consensus.cc:504] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.419804 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415a75c852814ea89b65d294d682c571" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:42.420102 4835 raft_consensus.cc:3037] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.423660 4835 raft_consensus.cc:2454] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:42.424257 4903 leader_election.cc:302] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:42.424795 5983 raft_consensus.cc:2781] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.424971 5983 raft_consensus.cc:686] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:42.425302 5983 consensus_queue.cc:237] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.436522 6495 raft_consensus.cc:482] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.436671 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415a75c852814ea89b65d294d682c571" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:42.436765 6495 raft_consensus.cc:504] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.436975 1944 raft_consensus.cc:3037] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.437485 6495 leader_election.cc:288] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:42.439520 5874 raft_consensus.cc:1050] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:12:42.440037 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df06752e8d7247dd962a376c629473c3" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:42.440253 4966 raft_consensus.cc:2454] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:42.440430 5874 consensus_queue.cc:237] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:42.440994 2138 leader_election.cc:302] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 184c543cb05d49cc98ba1c56617be7a5; no voters:
I20240503 08:12:42.441639 6495 raft_consensus.cc:2781] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.441648 1944 raft_consensus.cc:2454] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:42.441738 6495 raft_consensus.cc:482] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.441834 6495 raft_consensus.cc:3037] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.442327 6428 leader_election.cc:288] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:42.444715 6495 raft_consensus.cc:504] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.445986 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df06752e8d7247dd962a376c629473c3" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:42.446864 6495 leader_election.cc:288] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:42.446748 5786 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:42.448693 4966 raft_consensus.cc:3037] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.452419 2073 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:42.453787 4703 raft_consensus.cc:1261] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.453891 4966 raft_consensus.cc:2454] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:42.454475 5875 consensus_queue.cc:1035] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:42.455130 2138 leader_election.cc:302] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 184c543cb05d49cc98ba1c56617be7a5; no voters:
I20240503 08:12:42.455570 6495 raft_consensus.cc:2781] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.456560 6495 raft_consensus.cc:686] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:42.456938 6495 consensus_queue.cc:237] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.460346 2202 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:42.462823 1668 catalog_manager.cc:5484] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:42.463982 1944 raft_consensus.cc:1261] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.464996 5357 consensus_queue.cc:1035] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:42.466799 2202 raft_consensus.cc:1261] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.467619 5357 consensus_queue.cc:1035] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:42.479970 1672 catalog_manager.cc:5484] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:42.482955 4966 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:42.497764 5357 raft_consensus.cc:2932] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:42.500339 1672 catalog_manager.cc:5484] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:42.503420 2201 raft_consensus.cc:2932] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:42.504165 6406 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:42.503758 4703 raft_consensus.cc:2932] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:42.504653 6406 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:12:42.504945 6406 ts_tablet_manager.cc:1367] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.491s user 0.050s sys 0.010s
I20240503 08:12:42.505827 6406 raft_consensus.cc:348] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:42.506124 6406 raft_consensus.cc:729] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:12:42.506361 6406 consensus_queue.cc:260] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:42.512526 1665 catalog_manager.cc:5484] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:42.517988 6406 ts_tablet_manager.cc:1398] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20240503 08:12:42.519112 2224 tablet_copy_service.cc:342] P 184c543cb05d49cc98ba1c56617be7a5: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-913a3aacff964ac9a321fac187df33ee received from {username='slave'} at 127.27.157.199:47127
I20240503 08:12:42.519299 2224 tablet_copy_service.cc:433] P 184c543cb05d49cc98ba1c56617be7a5: ending tablet copy session f24648c92c384f1ba349eee801f985e2-913a3aacff964ac9a321fac187df33ee on tablet 913a3aacff964ac9a321fac187df33ee with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:12:42.530894 4835 consensus_queue.cc:237] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:42.401439 6587 ts_tablet_manager.cc:902] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
W20240503 08:12:42.535907 4903 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.157.199:39455, user_credentials={real_user=slave}} blocked reactor thread for 111680us
I20240503 08:12:42.535974 2074 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:42.537400 2201 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:42.539011 2074 consensus_queue.cc:237] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:42.539908 6588 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:42.543457 5874 raft_consensus.cc:1050] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:12:42.543736 6588 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:42.544322 5874 consensus_queue.cc:237] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:42.548295 5786 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:42.542198 2203 raft_consensus.cc:1261] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:42.549750 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54fe406b5f324d629243f7f939535620 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:42.550635 2074 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:42.550685 2203 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:42.552595 5357 consensus_queue.cc:1035] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:42.553460 1671 catalog_manager.cc:5484] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:42.555905 4701 raft_consensus.cc:1261] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:42.557121 5875 consensus_queue.cc:1035] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:42.558254 5875 raft_consensus.cc:2932] T 133911d44b394e24a89a5d9443230d32 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:42.362047 6577 tablet.cc:1722] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:42.372593 6094 raft_consensus.cc:482] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.560544 4946 tablet_service.cc:1514] Processing DeleteTablet for tablet 54fe406b5f324d629243f7f939535620 with delete_type TABLET_DATA_TOMBSTONED (TS 00b61625d8a7460892c118a2570afda5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:42.560626 6094 raft_consensus.cc:504] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.561458 6094 leader_election.cc:288] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:42.403286 6585 raft_consensus.cc:482] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:42.561679 6585 raft_consensus.cc:504] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.563050 6585 leader_election.cc:288] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:42.438596 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df06752e8d7247dd962a376c629473c3" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:42.563838 4569 raft_consensus.cc:2454] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:42.445911 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df06752e8d7247dd962a376c629473c3" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:42.564512 4568 raft_consensus.cc:3037] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.566442 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30] alter complete (version 1)
I20240503 08:12:42.567515 2203 raft_consensus.cc:2932] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:42.568013 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84831077defe434c9f29eecfae7b44c8" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:42.568140 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84831077defe434c9f29eecfae7b44c8" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:42.568207 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df06752e8d7247dd962a376c629473c3" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:42.568413 4965 raft_consensus.cc:2454] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:42.568477 4967 raft_consensus.cc:2379] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f95b05d111e4090bbc59dec394ed0ab in current term 1: Already voted for candidate 184c543cb05d49cc98ba1c56617be7a5 in this term.
I20240503 08:12:42.570360 4502 leader_election.cc:302] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:42.571589 1944 raft_consensus.cc:2932] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
W20240503 08:12:42.575160 4504 leader_election.cc:341] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:42.576453 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df06752e8d7247dd962a376c629473c3" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:42.577334 4502 leader_election.cc:302] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab; no voters: 00b61625d8a7460892c118a2570afda5, 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:42.577913 6094 raft_consensus.cc:2781] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:42.578069 6094 raft_consensus.cc:482] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:42.578176 6094 raft_consensus.cc:3037] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.579002 6551 tablet_bootstrap.cc:492] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:42.579232 6551 ts_tablet_manager.cc:1367] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.202s user 0.021s sys 0.000s
I20240503 08:12:42.580099 6551 raft_consensus.cc:348] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.580344 6551 raft_consensus.cc:374] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:42.580415 6551 raft_consensus.cc:729] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:42.579862 6466 tablet_copy_client.cc:308] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:42.581561 6094 raft_consensus.cc:504] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.582118 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 79579b65538d47f7973fa51f973b3820 from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:56593)
I20240503 08:12:42.582290 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 79579b65538d47f7973fa51f973b3820 from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:56593: session id = 5f95b05d111e4090bbc59dec394ed0ab-79579b65538d47f7973fa51f973b3820
I20240503 08:12:42.587975 1671 catalog_manager.cc:5484] T 133911d44b394e24a89a5d9443230d32 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:42.590358 2093 tablet_copy_source_session.cc:205] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:42.592800 4568 raft_consensus.cc:2454] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:42.593461 6551 consensus_queue.cc:260] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.593847 6585 raft_consensus.cc:2726] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:42.595422 6551 ts_tablet_manager.cc:1398] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240503 08:12:42.598639 6466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79579b65538d47f7973fa51f973b3820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.601534 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 133911d44b394e24a89a5d9443230d32 with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:42.601601 2203 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.609874 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84831077defe434c9f29eecfae7b44c8" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:42.610198 4967 raft_consensus.cc:3037] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.610917 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84831077defe434c9f29eecfae7b44c8" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:42.611178 4835 raft_consensus.cc:3037] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:42.613370 4967 raft_consensus.cc:2454] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:42.613956 4502 leader_election.cc:302] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:42.614661 6585 raft_consensus.cc:2781] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:42.614841 6585 raft_consensus.cc:686] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:42.615000 4835 raft_consensus.cc:2454] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:42.615150 6585 consensus_queue.cc:237] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:42.619172 6416 raft_consensus.cc:1050] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:42.620272 6416 consensus_queue.cc:237] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:42.625916 4568 raft_consensus.cc:1261] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.626940 6423 consensus_queue.cc:1035] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:42.628963 1943 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.629115 1942 raft_consensus.cc:1261] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.629885 6423 consensus_queue.cc:1035] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:42.630685 6094 leader_election.cc:288] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:42.627527 6604 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20240503 08:12:42.632838 5874 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240503 08:12:42.637924 6611 tablet_replica.cc:330] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:42.638355 6611 raft_consensus.cc:2229] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.638808 6611 raft_consensus.cc:2258] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.640244 6611 ts_tablet_manager.cc:1875] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:42.646173 6577 tablet.cc:1620] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:42.653785 6587 tablet_copy_client.cc:308] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:42.654827 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 53197deeb8d7439c90b019a6a90857e0 from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:43591)
I20240503 08:12:42.655078 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 53197deeb8d7439c90b019a6a90857e0 from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:43591: session id = 00b61625d8a7460892c118a2570afda5-53197deeb8d7439c90b019a6a90857e0
I20240503 08:12:42.658247 1964 tablet_copy_source_session.cc:205] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:42.660269 6587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53197deeb8d7439c90b019a6a90857e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:42.666843 6611 ts_tablet_manager.cc:1888] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:42.667461 6611 log.cc:1196] T 133911d44b394e24a89a5d9443230d32 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/133911d44b394e24a89a5d9443230d32
I20240503 08:12:42.668972 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 133911d44b394e24a89a5d9443230d32 (table rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]) successfully deleted
I20240503 08:12:42.670153 6423 raft_consensus.cc:2932] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:42.671022 6466 tablet_copy_client.cc:783] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:42.672140 6466 tablet_copy_client.cc:647] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:42.678740 1943 raft_consensus.cc:2932] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:42.679508 6587 tablet_copy_client.cc:783] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:42.679855 6587 tablet_copy_client.cc:647] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:42.688272 6587 tablet_copy_client.cc:523] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:42.689265 1668 catalog_manager.cc:5484] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:42.696269 6587 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:42.715615 6607 tablet_replica.cc:330] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:42.716094 6607 raft_consensus.cc:2229] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.716641 6607 raft_consensus.cc:2258] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.718127 6607 ts_tablet_manager.cc:1875] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 08:12:42.719208 2010 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.198:46559, user_credentials={real_user=slave}} blocked reactor thread for 156053us
W20240503 08:12:42.720647 2008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.196:42183, user_credentials={real_user=slave}} blocked reactor thread for 118431us
I20240503 08:12:42.726814 1672 catalog_manager.cc:5484] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:42.730342 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 133911d44b394e24a89a5d9443230d32 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:42.733196 1668 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:42.738695 6466 tablet_copy_client.cc:523] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:42.742424 6466 tablet_bootstrap.cc:492] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:42.764425 4835 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.767051 6466 tablet_bootstrap.cc:492] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:42.767043 4568 raft_consensus.cc:2932] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:42.767978 6466 tablet_bootstrap.cc:492] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:42.768445 6466 ts_tablet_manager.cc:1367] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.004s
I20240503 08:12:42.769316 4701 raft_consensus.cc:2932] T 133911d44b394e24a89a5d9443230d32 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:42.770059 6466 raft_consensus.cc:348] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:42.770591 6466 raft_consensus.cc:729] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:42.771257 6466 consensus_queue.cc:260] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:42.783190 1812 consensus_queue.cc:237] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:42.786058 2074 raft_consensus.cc:1261] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:42.795949 6578 tablet.cc:1722] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:42.796361 4568 raft_consensus.cc:1261] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:42.796521 6578 tablet.cc:1620] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:42.796958 6624 consensus_queue.cc:1035] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:42.797410 6609 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:42.799904 1944 raft_consensus.cc:1261] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:42.800053 6624 raft_consensus.cc:2932] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:42.800758 6423 consensus_queue.cc:1035] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:42.802951 1944 raft_consensus.cc:2932] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:42.810812 4568 raft_consensus.cc:2932] T 8fc8279b96f74e879b499b0143502c8f P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:42.814903 6466 ts_tablet_manager.cc:1398] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.046s user 0.003s sys 0.000s
I20240503 08:12:42.815279 1671 catalog_manager.cc:5484] T 8fc8279b96f74e879b499b0143502c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:42.819183 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30] alter complete (version 2)
I20240503 08:12:42.820327 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8fc8279b96f74e879b499b0143502c8f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:42.824074 1672 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet 1f7ed763c33a429682e5c6a44668cd48 (table rebalancer_dynamic_table [id=90b0088b227444a98e26dc6068effc30]). Expected version 2 got 1
W20240503 08:12:42.825240 6627 log.cc:924] Time spent T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240503 08:12:42.830643 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc8279b96f74e879b499b0143502c8f with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:42.832135 6577 tablet.cc:1722] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:42.832760 6577 tablet.cc:1620] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:42.835856 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-79579b65538d47f7973fa51f973b3820 received from {username='slave'} at 127.27.157.197:56593
I20240503 08:12:42.836138 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-79579b65538d47f7973fa51f973b3820 on tablet 79579b65538d47f7973fa51f973b3820 with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:42.793929 6581 tablet.cc:1722] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:42.837400 6581 tablet.cc:1620] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:42.822028 6587 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:42.825788 6428 consensus_queue.cc:1035] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.838505 6587 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:12:42.838881 6587 ts_tablet_manager.cc:1367] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.143s user 0.058s sys 0.012s
I20240503 08:12:42.840039 6587 raft_consensus.cc:348] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:42.840811 6587 raft_consensus.cc:729] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:12:42.841039 6587 consensus_queue.cc:260] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:42.841866 6587 ts_tablet_manager.cc:1398] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:12:42.842818 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-53197deeb8d7439c90b019a6a90857e0 received from {username='slave'} at 127.27.157.200:43591
I20240503 08:12:42.842970 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 00b61625d8a7460892c118a2570afda5-53197deeb8d7439c90b019a6a90857e0 on tablet 53197deeb8d7439c90b019a6a90857e0 with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:12:42.844265 5875 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:42.845844 4835 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:42.853657 1942 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:42.854611 2203 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:42.862658 1665 catalog_manager.cc:5484] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:42.870815 4834 raft_consensus.cc:1203] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:42.874219 2074 consensus_queue.cc:237] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:42.881258 2201 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:42.881878 5875 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:42.885269 5983 consensus_queue.cc:1035] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:42.889314 4834 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:42.890754 6609 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:42.893620 5875 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:42.895960 6635 tablet_replica.cc:330] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:42.896358 6635 raft_consensus.cc:2229] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.896867 6635 raft_consensus.cc:2258] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.897740 6635 ts_tablet_manager.cc:1875] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:42.898938 2201 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:42.906808 1665 catalog_manager.cc:5484] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:42.907251 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 044d8e5a69444865980ffc06a6be9080 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:42.881325 6607 ts_tablet_manager.cc:1888] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:42.912418 6607 log.cc:1196] T 54fe406b5f324d629243f7f939535620 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/54fe406b5f324d629243f7f939535620
I20240503 08:12:42.913565 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 54fe406b5f324d629243f7f939535620 (table rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]) successfully deleted
I20240503 08:12:42.913108 4834 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:42.927587 1672 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:42.931231 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 044d8e5a69444865980ffc06a6be9080 with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:42.932302 6611 tablet_replica.cc:330] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:42.932636 6611 raft_consensus.cc:2229] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:42.932971 6611 raft_consensus.cc:2258] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:42.953303 4967 raft_consensus.cc:1203] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:42.955572 6635 ts_tablet_manager.cc:1888] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 08:12:42.956099 6635 log.cc:1196] T 8fc8279b96f74e879b499b0143502c8f P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/8fc8279b96f74e879b499b0143502c8f
I20240503 08:12:42.956081 6611 ts_tablet_manager.cc:1875] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:42.963624 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 8fc8279b96f74e879b499b0143502c8f (table rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]) successfully deleted
I20240503 08:12:42.973230 6577 tablet.cc:1722] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:42.986629 6581 tablet.cc:1722] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:42.995996 6611 ts_tablet_manager.cc:1888] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:42.996182 6611 log.cc:1196] T 044d8e5a69444865980ffc06a6be9080 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/044d8e5a69444865980ffc06a6be9080
I20240503 08:12:42.999001 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 044d8e5a69444865980ffc06a6be9080 (table rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]) successfully deleted
I20240503 08:12:43.008543 6578 tablet.cc:1722] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:43.009112 6431 consensus_queue.cc:1035] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.014099 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=90b0088b227444a98e26dc6068effc30] alter complete (version 3)
I20240503 08:12:43.043895 6585 consensus_queue.cc:1035] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.056501 6495 consensus_queue.cc:1035] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.072362 6094 consensus_queue.cc:1035] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.129957 1665 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:43.130211 1665 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:43.134325 1665 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 1f7ed763c33a429682e5c6a44668cd48
I20240503 08:12:43.135673 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7ed763c33a429682e5c6a44668cd48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:43.135974 4549 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7ed763c33a429682e5c6a44668cd48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:43.136565 4946 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7ed763c33a429682e5c6a44668cd48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:43 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:43.136829 6657 tablet_replica.cc:330] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:43.136907 6607 tablet_replica.cc:330] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:43.137107 6657 raft_consensus.cc:2229] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:43.137140 6607 raft_consensus.cc:2229] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.137384 6607 raft_consensus.cc:2258] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.137940 6657 raft_consensus.cc:2258] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.138460 6656 tablet_replica.cc:330] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:43.138692 6607 ts_tablet_manager.cc:1875] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.138758 6656 raft_consensus.cc:2229] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.138974 6656 raft_consensus.cc:2258] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.139447 6657 ts_tablet_manager.cc:1875] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.140882 6656 ts_tablet_manager.cc:1875] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:43.148550 6657 ts_tablet_manager.cc:1888] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:43.148782 6657 log.cc:1196] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/1f7ed763c33a429682e5c6a44668cd48
I20240503 08:12:43.149300 6657 ts_tablet_manager.cc:1909] T 1f7ed763c33a429682e5c6a44668cd48 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:43.150599 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 1f7ed763c33a429682e5c6a44668cd48 (table rebalancer_dynamic_table_new_name [id=90b0088b227444a98e26dc6068effc30]) successfully deleted
I20240503 08:12:43.154430 6607 ts_tablet_manager.cc:1888] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:43.156884 6656 ts_tablet_manager.cc:1888] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:43.157205 6656 log.cc:1196] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/1f7ed763c33a429682e5c6a44668cd48
I20240503 08:12:43.157212 6607 log.cc:1196] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/1f7ed763c33a429682e5c6a44668cd48
I20240503 08:12:43.157790 6607 ts_tablet_manager.cc:1909] T 1f7ed763c33a429682e5c6a44668cd48 P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:43.157781 6656 ts_tablet_manager.cc:1909] T 1f7ed763c33a429682e5c6a44668cd48 P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:43.159073 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 1f7ed763c33a429682e5c6a44668cd48 (table rebalancer_dynamic_table_new_name [id=90b0088b227444a98e26dc6068effc30]) successfully deleted
I20240503 08:12:43.164966 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 1f7ed763c33a429682e5c6a44668cd48 (table rebalancer_dynamic_table_new_name [id=90b0088b227444a98e26dc6068effc30]) successfully deleted
I20240503 08:12:43.174522 4568 raft_consensus.cc:1203] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:43.237597 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:43.248586 4549 tablet_service.cc:1467] Processing CreateTablet for tablet bef84f42a3e344efb44c172ce88720db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=29e22a95084c481ab6e0b546faf34fad]), partition=
I20240503 08:12:43.248845 4814 tablet_service.cc:1467] Processing CreateTablet for tablet bef84f42a3e344efb44c172ce88720db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=29e22a95084c481ab6e0b546faf34fad]), partition=
I20240503 08:12:43.249049 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef84f42a3e344efb44c172ce88720db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.249198 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef84f42a3e344efb44c172ce88720db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.253947 2054 tablet_service.cc:1467] Processing CreateTablet for tablet bef84f42a3e344efb44c172ce88720db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=29e22a95084c481ab6e0b546faf34fad]), partition=
I20240503 08:12:43.254336 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef84f42a3e344efb44c172ce88720db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.263836 6682 tablet_bootstrap.cc:492] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:43.265142 6684 tablet_bootstrap.cc:492] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:43.267907 6684 tablet_bootstrap.cc:654] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.276434 6683 tablet_bootstrap.cc:492] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:43.283862 6682 tablet_bootstrap.cc:654] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.285763 6683 tablet_bootstrap.cc:654] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.294622 6684 tablet_bootstrap.cc:492] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:43.294934 6684 ts_tablet_manager.cc:1367] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.017s
I20240503 08:12:43.295084 6683 tablet_bootstrap.cc:492] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:43.295348 6683 ts_tablet_manager.cc:1367] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240503 08:12:43.295967 6684 raft_consensus.cc:348] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.296320 6684 raft_consensus.cc:374] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.296212 6683 raft_consensus.cc:348] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.296422 6683 raft_consensus.cc:374] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.296521 6683 raft_consensus.cc:729] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.296552 6684 raft_consensus.cc:729] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.296844 6683 consensus_queue.cc:260] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.296931 6684 consensus_queue.cc:260] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.298889 6683 ts_tablet_manager.cc:1398] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:43.301517 6684 ts_tablet_manager.cc:1398] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:43.312772 6682 tablet_bootstrap.cc:492] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:43.313060 6682 ts_tablet_manager.cc:1367] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.006s
I20240503 08:12:43.314146 6682 raft_consensus.cc:348] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.314352 6682 raft_consensus.cc:374] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.314466 6682 raft_consensus.cc:729] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.314724 6682 consensus_queue.cc:260] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.315392 6682 ts_tablet_manager.cc:1398] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:43.335178 6559 raft_consensus.cc:1050] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:12:43.336405 6559 consensus_queue.cc:237] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:43.337822 6495 raft_consensus.cc:1050] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:12:43.338709 6495 consensus_queue.cc:237] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:43.341465 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:43.346547 1942 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:43.346792 4835 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:43.349946 1812 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:43.350886 6431 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:43.351588 6431 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:43.353070 1812 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:43.354385 2204 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:43.355293 4965 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:43.355357 6375 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:43.355234 6647 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 08:12:43.359431 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1e2530d5f2431baf0b2dd41081bb9d (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=d79878310527447b9246772f2dcc1501]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.359970 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1e2530d5f2431baf0b2dd41081bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.360008 6375 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:43.360134 4549 tablet_service.cc:1467] Processing CreateTablet for tablet f567b4ad00e242ba94187bdaeda71d66 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=d79878310527447b9246772f2dcc1501]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.360613 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f567b4ad00e242ba94187bdaeda71d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.361004 6375 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:43.362299 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1e2530d5f2431baf0b2dd41081bb9d (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=d79878310527447b9246772f2dcc1501]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.362768 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1e2530d5f2431baf0b2dd41081bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.364400 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1e2530d5f2431baf0b2dd41081bb9d (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=d79878310527447b9246772f2dcc1501]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.364830 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1e2530d5f2431baf0b2dd41081bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.366803 1923 tablet_service.cc:1467] Processing CreateTablet for tablet f567b4ad00e242ba94187bdaeda71d66 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=d79878310527447b9246772f2dcc1501]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.367252 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f567b4ad00e242ba94187bdaeda71d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.380867 6707 tablet_bootstrap.cc:492] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:43.382892 4814 tablet_service.cc:1467] Processing CreateTablet for tablet f567b4ad00e242ba94187bdaeda71d66 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=d79878310527447b9246772f2dcc1501]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.383725 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f567b4ad00e242ba94187bdaeda71d66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.383736 4548 tablet_service.cc:1467] Processing CreateTablet for tablet eb4e33ffc60043d9a50f8381e85b927a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=d79878310527447b9246772f2dcc1501]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.384286 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4e33ffc60043d9a50f8381e85b927a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.389109 4683 tablet_service.cc:1467] Processing CreateTablet for tablet eb4e33ffc60043d9a50f8381e85b927a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=d79878310527447b9246772f2dcc1501]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.389732 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4e33ffc60043d9a50f8381e85b927a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.390640 6684 tablet_bootstrap.cc:492] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:43.391875 1793 tablet_service.cc:1467] Processing CreateTablet for tablet eb4e33ffc60043d9a50f8381e85b927a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=d79878310527447b9246772f2dcc1501]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:43.392441 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4e33ffc60043d9a50f8381e85b927a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:43.395982 6684 tablet_bootstrap.cc:654] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.415448 6682 tablet_bootstrap.cc:492] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:43.417855 6682 tablet_bootstrap.cc:654] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.436736 6710 tablet_bootstrap.cc:492] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:43.438779 6682 tablet_bootstrap.cc:492] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:43.439141 6682 ts_tablet_manager.cc:1367] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.006s
I20240503 08:12:43.440402 6682 raft_consensus.cc:348] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.442306 6684 tablet_bootstrap.cc:492] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:43.442727 6684 ts_tablet_manager.cc:1367] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.002s
I20240503 08:12:43.444568 6707 tablet_bootstrap.cc:654] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.444839 6682 raft_consensus.cc:374] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.445029 6682 raft_consensus.cc:729] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.445573 6682 consensus_queue.cc:260] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.446588 6708 tablet_bootstrap.cc:492] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:43.450102 6684 raft_consensus.cc:348] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.450703 6684 raft_consensus.cc:374] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.450882 6684 raft_consensus.cc:729] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.451510 6684 consensus_queue.cc:260] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.452930 6682 ts_tablet_manager.cc:1398] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 08:12:43.454475 6710 tablet_bootstrap.cc:654] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.457267 6684 ts_tablet_manager.cc:1398] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240503 08:12:43.460990 6684 tablet_bootstrap.cc:492] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
W20240503 08:12:43.464550 1878 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.200:45283, user_credentials={real_user=slave}} blocked reactor thread for 106615us
I20240503 08:12:43.468159 6700 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:43.473246 1942 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:43.479413 1813 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:43.481444 6707 tablet_bootstrap.cc:492] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:43.481618 6707 ts_tablet_manager.cc:1367] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.000s
I20240503 08:12:43.482484 6707 raft_consensus.cc:348] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.482698 6707 raft_consensus.cc:374] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.482791 6707 raft_consensus.cc:729] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.485488 6708 tablet_bootstrap.cc:654] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.486042 1668 catalog_manager.cc:5484] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:43.490386 6710 tablet_bootstrap.cc:492] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:43.493548 6710 ts_tablet_manager.cc:1367] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.001s
I20240503 08:12:43.490500 4834 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:43.494782 6710 raft_consensus.cc:348] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:43.495077 6710 raft_consensus.cc:374] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.495186 6710 raft_consensus.cc:729] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.501171 6684 tablet_bootstrap.cc:654] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.505203 6706 tablet_bootstrap.cc:492] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:43.507284 6706 tablet_bootstrap.cc:654] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.502988 6710 consensus_queue.cc:260] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:43.512113 2204 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:43.512967 2203 consensus_queue.cc:237] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:43.513576 6710 ts_tablet_manager.cc:1398] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.020s user 0.000s sys 0.003s
I20240503 08:12:43.492597 6707 consensus_queue.cc:260] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.507411 6375 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:43.518951 1813 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:43.519672 4834 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:43.520231 1665 catalog_manager.cc:5484] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:43.521286 6700 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:43.523546 1813 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:43.526239 6700 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:43.527580 4967 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:43.530424 6495 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:43.535140 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 913a3aacff964ac9a321fac187df33ee with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:43.536257 1942 consensus_queue.cc:237] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:43.537358 1813 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:43.540887 1668 catalog_manager.cc:5484] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:43.543889 4835 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:43.545361 1812 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:43.546494 6707 ts_tablet_manager.cc:1398] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.065s user 0.003s sys 0.000s
I20240503 08:12:43.546847 6707 tablet_bootstrap.cc:492] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:43.546789 6375 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:43.546978 6586 raft_consensus.cc:482] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:43.547165 6586 raft_consensus.cc:504] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.548108 6586 leader_election.cc:288] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:43.551833 6707 tablet_bootstrap.cc:654] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.551831 4967 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:43.550555 1812 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:43.552948 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f567b4ad00e242ba94187bdaeda71d66" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:43.553315 1944 raft_consensus.cc:2454] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:43.553992 4503 leader_election.cc:302] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:43.556492 6559 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:43.561035 6375 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:43.561581 6586 raft_consensus.cc:2781] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:43.561795 6586 raft_consensus.cc:482] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:43.561908 6586 raft_consensus.cc:3037] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.563227 4966 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:43.569327 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 913a3aacff964ac9a321fac187df33ee with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:43.576653 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f567b4ad00e242ba94187bdaeda71d66" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:43.578919 1671 catalog_manager.cc:5484] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:43.580776 4834 raft_consensus.cc:2454] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:43.583830 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 53197deeb8d7439c90b019a6a90857e0 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:43.575066 6586 raft_consensus.cc:504] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.586433 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53197deeb8d7439c90b019a6a90857e0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:43.586642 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f567b4ad00e242ba94187bdaeda71d66" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:43.586906 1942 raft_consensus.cc:3037] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.591675 1942 raft_consensus.cc:2454] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:43.593318 4503 leader_election.cc:302] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:43.593746 6506 raft_consensus.cc:2781] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:43.593892 6506 raft_consensus.cc:686] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:43.593998 6708 tablet_bootstrap.cc:492] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:43.594192 6506 consensus_queue.cc:237] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.595379 6708 ts_tablet_manager.cc:1367] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.148s user 0.004s sys 0.009s
I20240503 08:12:43.599426 6708 raft_consensus.cc:348] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:43.600885 6708 raft_consensus.cc:374] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.601095 6708 raft_consensus.cc:729] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.601486 6708 consensus_queue.cc:260] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:43.606823 6723 tablet_replica.cc:330] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:43.607108 6723 raft_consensus.cc:2229] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.607846 6725 tablet_replica.cc:330] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:43.608157 6725 raft_consensus.cc:2229] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.608486 6725 raft_consensus.cc:2258] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.608873 6708 ts_tablet_manager.cc:1398] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20240503 08:12:43.609678 6725 ts_tablet_manager.cc:1875] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:43.611050 5875 raft_consensus.cc:1050] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:43.611896 5875 consensus_queue.cc:237] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:43.610906 6586 leader_election.cc:288] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:43.617419 6708 tablet_bootstrap.cc:492] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:43.619536 6723 raft_consensus.cc:2258] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.619230 6708 tablet_bootstrap.cc:654] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:43.622030 6684 tablet_bootstrap.cc:492] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:43.622195 6684 ts_tablet_manager.cc:1367] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.161s user 0.008s sys 0.000s
I20240503 08:12:43.621883 1665 catalog_manager.cc:5484] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:43.622929 6723 ts_tablet_manager.cc:1875] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:43.623040 6684 raft_consensus.cc:348] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:43.626412 6684 raft_consensus.cc:374] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.626555 6684 raft_consensus.cc:729] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.624954 6707 tablet_bootstrap.cc:492] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:43.626948 6707 ts_tablet_manager.cc:1367] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.080s user 0.015s sys 0.019s
I20240503 08:12:43.628248 6706 tablet_bootstrap.cc:492] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:43.628443 6706 ts_tablet_manager.cc:1367] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.003s
I20240503 08:12:43.629456 6706 raft_consensus.cc:348] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:43.629689 6706 raft_consensus.cc:374] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.629796 6706 raft_consensus.cc:729] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.630146 6706 consensus_queue.cc:260] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:43.631800 6707 raft_consensus.cc:348] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:43.632066 6707 raft_consensus.cc:374] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.632150 6707 raft_consensus.cc:729] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.632390 6707 consensus_queue.cc:260] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:43.634529 4568 raft_consensus.cc:1261] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:43.634709 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f567b4ad00e242ba94187bdaeda71d66" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:43.634971 4834 raft_consensus.cc:3037] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.635416 6612 consensus_queue.cc:1035] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:43.637406 6706 ts_tablet_manager.cc:1398] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 08:12:43.640156 4834 raft_consensus.cc:2454] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:43.647828 4703 raft_consensus.cc:1261] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:43.649473 6612 consensus_queue.cc:1035] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:43.652880 1812 raft_consensus.cc:1261] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:43.653734 5875 consensus_queue.cc:1035] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:43.655995 6684 consensus_queue.cc:260] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:43.662042 6684 ts_tablet_manager.cc:1398] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.040s user 0.002s sys 0.001s
I20240503 08:12:43.678017 6725 ts_tablet_manager.cc:1888] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:43.678225 6725 log.cc:1196] T 53197deeb8d7439c90b019a6a90857e0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/53197deeb8d7439c90b019a6a90857e0
I20240503 08:12:43.678894 6707 ts_tablet_manager.cc:1398] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20240503 08:12:43.679301 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 53197deeb8d7439c90b019a6a90857e0 (table rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]) successfully deleted
I20240503 08:12:43.680508 6708 tablet_bootstrap.cc:492] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:43.680769 6708 ts_tablet_manager.cc:1367] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.008s
I20240503 08:12:43.681071 6541 raft_consensus.cc:482] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:43.681237 6541 raft_consensus.cc:504] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:43.681640 6708 raft_consensus.cc:348] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:43.681879 6708 raft_consensus.cc:374] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:43.681955 6708 raft_consensus.cc:729] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:43.682034 6541 leader_election.cc:288] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:43.683172 6612 raft_consensus.cc:482] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:43.683272 6723 ts_tablet_manager.cc:1888] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:43.683393 6612 raft_consensus.cc:504] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.683506 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4e33ffc60043d9a50f8381e85b927a" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:43.684194 6612 leader_election.cc:288] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:43.686259 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef84f42a3e344efb44c172ce88720db" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:43.686511 4569 raft_consensus.cc:2454] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:43.683473 6723 log.cc:1196] T 913a3aacff964ac9a321fac187df33ee P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/913a3aacff964ac9a321fac187df33ee
I20240503 08:12:43.687350 2007 leader_election.cc:302] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:43.687981 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 913a3aacff964ac9a321fac187df33ee (table rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]) successfully deleted
I20240503 08:12:43.688344 6612 raft_consensus.cc:2781] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:43.688351 6708 consensus_queue.cc:260] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:43.688531 6612 raft_consensus.cc:482] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:43.688766 6612 raft_consensus.cc:3037] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.688925 1813 raft_consensus.cc:2454] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:43.692004 4637 leader_election.cc:302] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:43.692610 6541 raft_consensus.cc:2781] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:43.692754 6541 raft_consensus.cc:482] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:43.692836 6541 raft_consensus.cc:3037] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.699120 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef84f42a3e344efb44c172ce88720db" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:43.699509 4835 raft_consensus.cc:2454] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:43.703451 6541 raft_consensus.cc:504] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:43.704978 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4e33ffc60043d9a50f8381e85b927a" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:43.705332 6541 leader_election.cc:288] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:43.705368 4569 raft_consensus.cc:3037] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.705797 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4e33ffc60043d9a50f8381e85b927a" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:43.705802 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4e33ffc60043d9a50f8381e85b927a" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:43.706094 1813 raft_consensus.cc:3037] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.709991 4569 raft_consensus.cc:2454] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:43.710014 1813 raft_consensus.cc:2454] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:43.710742 4635 leader_election.cc:302] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:43.711365 6541 raft_consensus.cc:2781] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:43.713177 6612 raft_consensus.cc:504] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.714792 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef84f42a3e344efb44c172ce88720db" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:43.715291 4569 raft_consensus.cc:3037] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.720075 4569 raft_consensus.cc:2454] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:43.720944 2007 leader_election.cc:302] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:43.721881 6609 raft_consensus.cc:2781] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:43.722034 6609 raft_consensus.cc:686] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:43.722338 6609 consensus_queue.cc:237] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:43.756410 6624 raft_consensus.cc:482] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:43.756603 6624 raft_consensus.cc:504] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:43.757305 6624 leader_election.cc:288] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:43.758103 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1e2530d5f2431baf0b2dd41081bb9d" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:43.758358 4965 raft_consensus.cc:2454] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:43.758863 1747 leader_election.cc:302] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:43.759258 6624 raft_consensus.cc:2781] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:43.759395 6624 raft_consensus.cc:482] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:43.759477 6624 raft_consensus.cc:3037] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.762776 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1e2530d5f2431baf0b2dd41081bb9d" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:43.763051 1942 raft_consensus.cc:2454] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:43.764930 6624 raft_consensus.cc:504] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:43.766188 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1e2530d5f2431baf0b2dd41081bb9d" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:43.766415 4965 raft_consensus.cc:3037] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.770630 4965 raft_consensus.cc:2454] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:43.771592 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1e2530d5f2431baf0b2dd41081bb9d" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:43.771852 1942 raft_consensus.cc:3037] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.774837 1942 raft_consensus.cc:2454] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:43.775375 1748 leader_election.cc:302] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:43.775388 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef84f42a3e344efb44c172ce88720db" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:43.775696 4835 raft_consensus.cc:3037] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:43.776257 6624 leader_election.cc:288] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:43.776762 6624 raft_consensus.cc:2781] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:43.777010 6624 raft_consensus.cc:686] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:43.777457 6624 consensus_queue.cc:237] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:43.779471 4835 raft_consensus.cc:2454] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:43.781219 6612 leader_election.cc:288] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
W20240503 08:12:43.808447 2009 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.194:41975, user_credentials={real_user=slave}} blocked reactor thread for 83299us
I20240503 08:12:43.817767 6744 raft_consensus.cc:2932] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:43.820703 6541 raft_consensus.cc:686] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:43.821097 6541 consensus_queue.cc:237] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:43.826277 4703 raft_consensus.cc:2932] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:43.830178 6708 ts_tablet_manager.cc:1398] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.149s user 0.025s sys 0.029s
I20240503 08:12:43.831844 1672 catalog_manager.cc:5484] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:43.833264 1665 catalog_manager.cc:5484] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:43.846158 1812 raft_consensus.cc:2932] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:43.848193 4568 raft_consensus.cc:2932] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:43.854169 1671 catalog_manager.cc:5484] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:43.859544 1665 catalog_manager.cc:5484] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:43.869860 2074 consensus_queue.cc:237] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:43.872417 4568 raft_consensus.cc:1261] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:43.873083 1812 raft_consensus.cc:1261] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:43.873059 6739 consensus_queue.cc:1035] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:43.873643 6609 consensus_queue.cc:1035] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:43.876673 6739 raft_consensus.cc:2932] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:43.878720 1812 raft_consensus.cc:2932] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:43.881477 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 79579b65538d47f7973fa51f973b3820 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:43.885603 1672 catalog_manager.cc:5484] T 79579b65538d47f7973fa51f973b3820 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:43.886047 4568 raft_consensus.cc:2932] T 79579b65538d47f7973fa51f973b3820 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:43.893328 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 79579b65538d47f7973fa51f973b3820 with delete_type TABLET_DATA_TOMBSTONED (TS 31f41542452b4b1eb994ff4223d039fc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:43.903892 6757 tablet_replica.cc:330] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:43.904145 6757 raft_consensus.cc:2229] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:43.904428 6757 raft_consensus.cc:2258] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:43.905540 6757 ts_tablet_manager.cc:1875] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:43.910821 6757 ts_tablet_manager.cc:1888] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:43.910969 6757 log.cc:1196] T 79579b65538d47f7973fa51f973b3820 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/79579b65538d47f7973fa51f973b3820
I20240503 08:12:43.912469 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 79579b65538d47f7973fa51f973b3820 (table rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]) successfully deleted
I20240503 08:12:43.922683 1942 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "733bed28775e4779b74e92d8a395234f"
dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57082
I20240503 08:12:43.922988 1942 raft_consensus.cc:595] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:43.957988 1672 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:43.966269 4568 raft_consensus.cc:1261] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:43.966706 1812 consensus_queue.cc:237] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:43.966969 6609 consensus_queue.cc:1035] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.969128 4835 raft_consensus.cc:1261] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:43.970083 6609 consensus_queue.cc:1035] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:43.973974 2203 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:43.975664 1749 consensus_peers.cc:485] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fede923140fb45a5a4f93c0846b62595. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:43.976944 6749 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:43.981809 2074 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:43.984738 6749 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:43.989864 6624 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:43.990679 2203 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:43.999256 1668 catalog_manager.cc:5484] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:44.002776 1672 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet bef84f42a3e344efb44c172ce88720db (table rebalancer_dynamic_table [id=29e22a95084c481ab6e0b546faf34fad]). Expected version 1 got 0
I20240503 08:12:44.005290 6789 tablet.cc:1722] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:44.006039 6789 tablet.cc:1620] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:44.021941 6788 tablet.cc:1722] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:44.022233 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=29e22a95084c481ab6e0b546faf34fad] alter complete (version 1)
I20240503 08:12:44.026173 1665 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet bef84f42a3e344efb44c172ce88720db (table rebalancer_dynamic_table [id=29e22a95084c481ab6e0b546faf34fad]). Expected version 1 got 0
I20240503 08:12:44.026945 1664 catalog_manager.cc:5576] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117) does not have the latest schema for tablet bef84f42a3e344efb44c172ce88720db (table rebalancer_dynamic_table [id=29e22a95084c481ab6e0b546faf34fad]). Expected version 1 got 0
I20240503 08:12:44.029755 6788 tablet.cc:1620] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: MemRowSet was empty: no flush needed.
I20240503 08:12:44.038789 2074 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:44.052220 6791 tablet.cc:1722] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:44.052893 6791 tablet.cc:1620] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:44.054834 1668 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet bef84f42a3e344efb44c172ce88720db (table rebalancer_dynamic_table [id=29e22a95084c481ab6e0b546faf34fad]). Expected version 1 got 0
I20240503 08:12:44.087500 1812 consensus_queue.cc:237] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:44.090885 6506 consensus_queue.cc:1035] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.092623 2203 raft_consensus.cc:1261] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:44.092759 1746 consensus_peers.cc:485] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e371231b34f45dfac455bbbbe156263. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:44.093593 2074 raft_consensus.cc:1261] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:44.094632 6749 consensus_queue.cc:1035] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:44.095225 6749 consensus_queue.cc:1035] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:44.111024 6586 consensus_queue.cc:1035] T f567b4ad00e242ba94187bdaeda71d66 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.113903 6735 raft_consensus.cc:2932] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:44.120290 2073 raft_consensus.cc:2932] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:44.127853 2203 raft_consensus.cc:2932] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:44.132920 1664 catalog_manager.cc:5484] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:44.135790 1668 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:44.149555 6788 tablet.cc:1722] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:44.150291 6788 tablet.cc:1620] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: MemRowSet was empty: no flush needed.
I20240503 08:12:44.152676 6791 tablet.cc:1722] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:44.152740 6789 tablet.cc:1722] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:44.153211 6791 tablet.cc:1620] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:44.153328 6789 tablet.cc:1620] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:44.174439 1671 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=29e22a95084c481ab6e0b546faf34fad] alter complete (version 2)
I20240503 08:12:44.181478 6559 raft_consensus.cc:979] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: : Instructing follower 00b61625d8a7460892c118a2570afda5 to start an election
I20240503 08:12:44.181751 6559 raft_consensus.cc:1067] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Signalling peer 00b61625d8a7460892c118a2570afda5 to start an election
I20240503 08:12:44.183279 4965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "733bed28775e4779b74e92d8a395234f"
dest_uuid: "00b61625d8a7460892c118a2570afda5"
from {username='slave'} at 127.27.157.194:41645
I20240503 08:12:44.183692 4965 raft_consensus.cc:482] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:44.183842 4965 raft_consensus.cc:3037] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:44.188177 4965 raft_consensus.cc:504] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:44.190132 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733bed28775e4779b74e92d8a395234f" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:44.190378 1812 raft_consensus.cc:3037] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:44.190505 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733bed28775e4779b74e92d8a395234f" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:44.190784 2203 raft_consensus.cc:3037] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:44.193040 4965 leader_election.cc:288] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:44.194474 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733bed28775e4779b74e92d8a395234f" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:44.194739 1944 raft_consensus.cc:3032] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:44.194816 1944 raft_consensus.cc:729] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:44.195175 1944 consensus_queue.cc:260] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:44.195847 1944 raft_consensus.cc:3037] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:44.196311 2203 raft_consensus.cc:2454] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 2.
I20240503 08:12:44.198763 1944 raft_consensus.cc:2454] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 2.
I20240503 08:12:44.199512 4902 leader_election.cc:302] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:44.199867 6593 raft_consensus.cc:2781] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:44.199992 6593 raft_consensus.cc:686] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:44.200332 6593 consensus_queue.cc:237] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:44.193496 1812 raft_consensus.cc:2454] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 2.
I20240503 08:12:44.209889 1671 catalog_manager.cc:5484] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 1 to 2, leader changed from 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:44.211417 6735 consensus_queue.cc:1035] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.233740 6624 consensus_queue.cc:1035] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.242977 4834 consensus_queue.cc:237] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:44.247921 4966 raft_consensus.cc:1261] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:12:44.248323 4769 consensus_peers.cc:485] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6658d7bcd634abd808637e474018965. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:44.248914 2073 raft_consensus.cc:1261] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.249658 6729 consensus_queue.cc:1035] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:44.254402 6541 consensus_queue.cc:1035] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.256440 6729 consensus_queue.cc:1035] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:44.261432 6729 raft_consensus.cc:2932] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:44.265391 4966 raft_consensus.cc:2932] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:44.289790 1664 catalog_manager.cc:5484] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:44.297070 1671 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:44.298156 6737 consensus_queue.cc:1035] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:44.298897 2073 raft_consensus.cc:2932] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:44.319221 6788 tablet.cc:1722] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:44.320896 6791 tablet.cc:1722] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:44.330054 6789 tablet.cc:1722] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 08:12:44.344146 2139 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.193:41181, user_credentials={real_user=slave}} blocked reactor thread for 59352.6us
I20240503 08:12:44.350848 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=29e22a95084c481ab6e0b546faf34fad] alter complete (version 3)
I20240503 08:12:44.371042 1944 consensus_queue.cc:237] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:44.373800 1812 raft_consensus.cc:1261] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:44.373894 1878 consensus_peers.cc:485] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa111e798e3f46d2a5bc1eebfe28f8bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:44.374262 2203 raft_consensus.cc:1261] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:44.374471 6733 consensus_queue.cc:1035] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:44.374893 6375 consensus_queue.cc:1035] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:44.378218 6375 raft_consensus.cc:2932] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:44.379011 1812 raft_consensus.cc:2932] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:44.379447 2203 raft_consensus.cc:2932] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:44.385628 1665 catalog_manager.cc:5484] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:44.415781 6838 ts_tablet_manager.cc:902] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:44.416664 4965 consensus_queue.cc:237] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
W20240503 08:12:44.420099 4903 consensus_peers.cc:485] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dab60fefb4704ef19490fb3c76bf47ab. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:44.420382 4569 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.420751 6838 tablet_copy_client.cc:308] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:41181
I20240503 08:12:44.421072 4834 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:44.421061 6808 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:44.421532 1833 tablet_copy_service.cc:140] P b8f581390da449ccb9e299011296405e: Received BeginTabletCopySession request for tablet fede923140fb45a5a4f93c0846b62595 from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:51373)
I20240503 08:12:44.421727 1833 tablet_copy_service.cc:161] P b8f581390da449ccb9e299011296405e: Beginning new tablet copy session on tablet fede923140fb45a5a4f93c0846b62595 from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:51373: session id = 31f41542452b4b1eb994ff4223d039fc-fede923140fb45a5a4f93c0846b62595
I20240503 08:12:44.422061 6808 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:44.424337 1833 tablet_copy_source_session.cc:205] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:44.424569 6808 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:44.426009 6838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fede923140fb45a5a4f93c0846b62595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.428314 4569 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:44.434016 6838 tablet_copy_client.cc:783] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:44.435768 6838 tablet_copy_client.cc:647] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:44.439249 1664 catalog_manager.cc:5484] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index -1 to 2, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.440143 6838 tablet_copy_client.cc:523] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:44.440415 4834 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:44.444161 6838 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:44.459133 1664 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:44.459604 1664 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:44.463364 1664 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet bef84f42a3e344efb44c172ce88720db
I20240503 08:12:44.463349 1944 consensus_queue.cc:237] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:44.464272 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet bef84f42a3e344efb44c172ce88720db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:44.464959 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet bef84f42a3e344efb44c172ce88720db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:44.465729 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet bef84f42a3e344efb44c172ce88720db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:44 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:44.475942 6855 tablet_replica.cc:330] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:44.476707 6854 tablet_replica.cc:330] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:44.477136 6855 raft_consensus.cc:2229] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.478300 6854 raft_consensus.cc:2229] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:44.479959 6855 raft_consensus.cc:2258] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.480895 2203 raft_consensus.cc:1261] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:44.481550 1877 consensus_peers.cc:485] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fba1e1952da44c0e956be1ef7f592f08. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:44.482169 6852 tablet_replica.cc:330] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:44.482647 6733 consensus_queue.cc:1035] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:44.484331 1664 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:44.486378 6854 raft_consensus.cc:2258] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.486692 6852 raft_consensus.cc:2229] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:44.486850 6855 ts_tablet_manager.cc:1875] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.489792 6852 raft_consensus.cc:2258] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:44.490773 2073 raft_consensus.cc:1261] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:44.491456 6852 ts_tablet_manager.cc:1875] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.491534 6733 consensus_queue.cc:1035] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:44.495605 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 275fc5bbb1494306b1139b7f950ef80f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=01490def2fa84e8ab1ccc8f8499d37cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.495764 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 275fc5bbb1494306b1139b7f950ef80f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=01490def2fa84e8ab1ccc8f8499d37cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.496080 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275fc5bbb1494306b1139b7f950ef80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.496259 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275fc5bbb1494306b1139b7f950ef80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.499891 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 09ae3cc0ce534128b49f2da744d23144 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=01490def2fa84e8ab1ccc8f8499d37cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.500399 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ae3cc0ce534128b49f2da744d23144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.500634 4549 tablet_service.cc:1467] Processing CreateTablet for tablet 09ae3cc0ce534128b49f2da744d23144 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=01490def2fa84e8ab1ccc8f8499d37cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.501076 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ae3cc0ce534128b49f2da744d23144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.502264 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 275fc5bbb1494306b1139b7f950ef80f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=01490def2fa84e8ab1ccc8f8499d37cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.502702 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275fc5bbb1494306b1139b7f950ef80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.503800 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 09ae3cc0ce534128b49f2da744d23144 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=01490def2fa84e8ab1ccc8f8499d37cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.504256 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ae3cc0ce534128b49f2da744d23144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.507033 4813 tablet_service.cc:1467] Processing CreateTablet for tablet 6427045c1be84157bfbc066642ff6505 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=01490def2fa84e8ab1ccc8f8499d37cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.511840 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6427045c1be84157bfbc066642ff6505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.516839 6838 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:44.517441 6838 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:12:44.517855 6838 ts_tablet_manager.cc:1367] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.074s user 0.056s sys 0.002s
I20240503 08:12:44.518884 6838 raft_consensus.cc:348] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:44.519345 6838 raft_consensus.cc:729] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:12:44.519642 6838 consensus_queue.cc:260] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:44.520458 6838 ts_tablet_manager.cc:1398] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:44.522269 1833 tablet_copy_service.cc:342] P b8f581390da449ccb9e299011296405e: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-fede923140fb45a5a4f93c0846b62595 received from {username='slave'} at 127.27.157.198:51373
I20240503 08:12:44.522413 1833 tablet_copy_service.cc:433] P b8f581390da449ccb9e299011296405e: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-fede923140fb45a5a4f93c0846b62595 on tablet fede923140fb45a5a4f93c0846b62595 with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:44.523962 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 6427045c1be84157bfbc066642ff6505 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=01490def2fa84e8ab1ccc8f8499d37cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.529078 6861 tablet_bootstrap.cc:492] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:44.529588 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6427045c1be84157bfbc066642ff6505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.530593 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 6427045c1be84157bfbc066642ff6505 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=01490def2fa84e8ab1ccc8f8499d37cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:44.531157 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6427045c1be84157bfbc066642ff6505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.531836 6861 tablet_bootstrap.cc:654] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.536633 6854 ts_tablet_manager.cc:1875] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:44.552289 6864 tablet_bootstrap.cc:492] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:44.554060 6860 tablet_bootstrap.cc:492] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:44.556509 6864 tablet_bootstrap.cc:654] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.558234 6375 raft_consensus.cc:2932] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:44.559422 6863 tablet_bootstrap.cc:492] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:44.561013 6858 tablet_bootstrap.cc:492] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:44.565676 2074 raft_consensus.cc:2932] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:44.567003 6855 ts_tablet_manager.cc:1888] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:44.567150 6855 log.cc:1196] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/bef84f42a3e344efb44c172ce88720db
I20240503 08:12:44.567652 6855 ts_tablet_manager.cc:1909] T bef84f42a3e344efb44c172ce88720db P f24648c92c384f1ba349eee801f985e2: Deleting consensus metadata
I20240503 08:12:44.569005 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet bef84f42a3e344efb44c172ce88720db (table rebalancer_dynamic_table_new_name [id=29e22a95084c481ab6e0b546faf34fad]) successfully deleted
I20240503 08:12:44.573875 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:44.568286 1664 catalog_manager.cc:5484] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.585018 6859 tablet_bootstrap.cc:492] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:44.586078 4681 tablet_service.cc:1467] Processing CreateTablet for tablet 07c97a95047e41f9ac878d4755403e98 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1aeae3adb01442938d48df7fa54e626d]), partition=
I20240503 08:12:44.586491 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c97a95047e41f9ac878d4755403e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.587989 6859 tablet_bootstrap.cc:654] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.591701 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 07c97a95047e41f9ac878d4755403e98 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1aeae3adb01442938d48df7fa54e626d]), partition=
I20240503 08:12:44.592145 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c97a95047e41f9ac878d4755403e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.593024 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 07c97a95047e41f9ac878d4755403e98 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1aeae3adb01442938d48df7fa54e626d]), partition=
I20240503 08:12:44.593477 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c97a95047e41f9ac878d4755403e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.603410 2204 raft_consensus.cc:2932] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:44.604753 6864 tablet_bootstrap.cc:492] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:44.604943 6864 ts_tablet_manager.cc:1367] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.011s
I20240503 08:12:44.607190 6858 tablet_bootstrap.cc:654] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.609807 6852 ts_tablet_manager.cc:1888] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:44.610072 6852 log.cc:1196] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/bef84f42a3e344efb44c172ce88720db
I20240503 08:12:44.610389 6864 raft_consensus.cc:348] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.611128 6863 tablet_bootstrap.cc:654] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.611887 6864 raft_consensus.cc:374] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.612529 6864 raft_consensus.cc:729] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.613142 6854 ts_tablet_manager.cc:1888] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:44.613617 6864 consensus_queue.cc:260] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.616017 6860 tablet_bootstrap.cc:654] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.617218 6854 log.cc:1196] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/bef84f42a3e344efb44c172ce88720db
I20240503 08:12:44.617969 6864 ts_tablet_manager.cc:1398] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
I20240503 08:12:44.617815 6854 ts_tablet_manager.cc:1909] T bef84f42a3e344efb44c172ce88720db P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:44.620975 6864 tablet_bootstrap.cc:492] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:44.621440 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet bef84f42a3e344efb44c172ce88720db (table rebalancer_dynamic_table_new_name [id=29e22a95084c481ab6e0b546faf34fad]) successfully deleted
I20240503 08:12:44.621831 6861 tablet_bootstrap.cc:492] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:44.610713 6852 ts_tablet_manager.cc:1909] T bef84f42a3e344efb44c172ce88720db P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:44.624668 6864 tablet_bootstrap.cc:654] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.624644 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet bef84f42a3e344efb44c172ce88720db (table rebalancer_dynamic_table_new_name [id=29e22a95084c481ab6e0b546faf34fad]) successfully deleted
I20240503 08:12:44.625222 6861 ts_tablet_manager.cc:1367] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.096s user 0.010s sys 0.031s
I20240503 08:12:44.626645 6861 raft_consensus.cc:348] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:44.629046 6861 raft_consensus.cc:374] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.629642 6861 raft_consensus.cc:729] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.633752 6858 tablet_bootstrap.cc:492] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:44.634083 6858 ts_tablet_manager.cc:1367] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.079s user 0.006s sys 0.000s
I20240503 08:12:44.635100 6858 raft_consensus.cc:348] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.635540 6858 raft_consensus.cc:374] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.635442 6861 consensus_queue.cc:260] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:44.636077 6858 raft_consensus.cc:729] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.636457 6858 consensus_queue.cc:260] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.638672 6863 tablet_bootstrap.cc:492] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:44.638860 6863 ts_tablet_manager.cc:1367] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.000s
I20240503 08:12:44.640662 2204 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:12:44.641402 6861 ts_tablet_manager.cc:1398] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 08:12:44.641340 6863 raft_consensus.cc:348] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:44.641487 6808 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:44.641806 6506 raft_consensus.cc:482] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.642403 6506 raft_consensus.cc:504] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.643249 6506 leader_election.cc:288] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:44.644274 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ae3cc0ce534128b49f2da744d23144" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:44.644529 4965 raft_consensus.cc:2454] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:44.645123 4502 leader_election.cc:302] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:44.645741 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ae3cc0ce534128b49f2da744d23144" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
W20240503 08:12:44.646512 4503 leader_election.cc:341] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:44.647022 6506 raft_consensus.cc:2781] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.647147 6506 raft_consensus.cc:482] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.647253 6506 raft_consensus.cc:3037] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.650363 6506 raft_consensus.cc:504] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.653208 6506 leader_election.cc:288] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:44.653362 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ae3cc0ce534128b49f2da744d23144" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:44.653683 4965 raft_consensus.cc:3037] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.653659 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ae3cc0ce534128b49f2da744d23144" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:44.655920 6863 raft_consensus.cc:374] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.656555 6863 raft_consensus.cc:729] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.656980 6863 consensus_queue.cc:260] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:44.660439 6861 tablet_bootstrap.cc:492] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
W20240503 08:12:44.660573 4503 leader_election.cc:341] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:44.663717 6863 ts_tablet_manager.cc:1398] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20240503 08:12:44.663787 6861 tablet_bootstrap.cc:654] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.667347 4965 raft_consensus.cc:2454] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:44.668108 4502 leader_election.cc:302] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters: b8f581390da449ccb9e299011296405e
I20240503 08:12:44.668818 6586 raft_consensus.cc:2781] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.668974 6586 raft_consensus.cc:686] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:44.669301 6586 consensus_queue.cc:237] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.671293 6864 tablet_bootstrap.cc:492] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:44.671702 6860 tablet_bootstrap.cc:492] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:44.671940 6860 ts_tablet_manager.cc:1367] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.118s user 0.005s sys 0.001s
I20240503 08:12:44.672946 6860 raft_consensus.cc:348] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:44.673269 6860 raft_consensus.cc:374] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.673425 6860 raft_consensus.cc:729] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.673832 6860 consensus_queue.cc:260] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:44.674643 6860 ts_tablet_manager.cc:1398] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:44.677474 6863 tablet_bootstrap.cc:492] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:44.675316 6860 tablet_bootstrap.cc:492] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:44.678319 6864 ts_tablet_manager.cc:1367] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.003s
I20240503 08:12:44.680090 6860 tablet_bootstrap.cc:654] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.682719 6859 tablet_bootstrap.cc:492] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:44.682997 6859 ts_tablet_manager.cc:1367] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.098s user 0.009s sys 0.005s
I20240503 08:12:44.683986 6864 raft_consensus.cc:348] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.684583 6858 ts_tablet_manager.cc:1398] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.050s user 0.000s sys 0.003s
I20240503 08:12:44.685596 6864 raft_consensus.cc:374] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.685259 6859 raft_consensus.cc:348] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.685905 6859 raft_consensus.cc:374] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.686004 6858 tablet_bootstrap.cc:492] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:44.686038 6864 raft_consensus.cc:729] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.686453 6863 tablet_bootstrap.cc:654] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.687232 6859 raft_consensus.cc:729] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.687584 6867 ts_tablet_manager.cc:902] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:44.687856 6859 consensus_queue.cc:260] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.689106 6859 ts_tablet_manager.cc:1398] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:12:44.687609 6864 consensus_queue.cc:260] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.698508 6864 ts_tablet_manager.cc:1398] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 08:12:44.699060 6880 tablet_bootstrap.cc:492] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:44.700731 6858 tablet_bootstrap.cc:654] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.701841 6880 tablet_bootstrap.cc:654] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:44.704509 6861 tablet_bootstrap.cc:492] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:44.704731 6861 ts_tablet_manager.cc:1367] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20240503 08:12:44.705605 6861 raft_consensus.cc:348] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.705847 6861 raft_consensus.cc:374] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.705929 6861 raft_consensus.cc:729] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.706198 6861 consensus_queue.cc:260] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.706755 6861 ts_tablet_manager.cc:1398] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:44.709203 6863 tablet_bootstrap.cc:492] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:44.709533 6863 ts_tablet_manager.cc:1367] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240503 08:12:44.710460 6863 raft_consensus.cc:348] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:44.711161 6863 raft_consensus.cc:374] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.711315 6863 raft_consensus.cc:729] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.712785 6860 tablet_bootstrap.cc:492] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:44.713058 6860 ts_tablet_manager.cc:1367] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.005s
I20240503 08:12:44.714136 6860 raft_consensus.cc:348] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:44.714473 6860 raft_consensus.cc:374] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.714843 6860 raft_consensus.cc:729] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.715123 6863 consensus_queue.cc:260] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:44.716418 1671 catalog_manager.cc:5484] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:44.722285 6863 ts_tablet_manager.cc:1398] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 08:12:44.715585 6860 consensus_queue.cc:260] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:44.722982 6860 ts_tablet_manager.cc:1398] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 08:12:44.729298 6858 tablet_bootstrap.cc:492] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:44.729605 6858 ts_tablet_manager.cc:1367] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.004s
I20240503 08:12:44.730998 6858 raft_consensus.cc:348] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:44.731637 6858 raft_consensus.cc:374] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.732334 6858 raft_consensus.cc:729] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.732812 6858 consensus_queue.cc:260] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:44.733512 6858 ts_tablet_manager.cc:1398] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:44.745661 1944 consensus_queue.cc:237] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:44.748435 1942 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:12:44.749310 6808 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:44.757513 6735 raft_consensus.cc:482] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.757766 6735 raft_consensus.cc:504] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.758798 6735 leader_election.cc:288] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:44.758867 1813 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:12:44.759907 6593 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:44.760069 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ae3cc0ce534128b49f2da744d23144" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:44.760483 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ae3cc0ce534128b49f2da744d23144" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:44.760768 4965 raft_consensus.cc:2379] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8f581390da449ccb9e299011296405e in current term 1: Already voted for candidate 5f95b05d111e4090bbc59dec394ed0ab in this term.
I20240503 08:12:44.762895 1747 leader_election.cc:302] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f581390da449ccb9e299011296405e; no voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:44.763693 6624 raft_consensus.cc:3037] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.766870 6624 raft_consensus.cc:2726] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:44.767922 1812 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:44.768558 6375 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:12:44.771162 1878 consensus_peers.cc:485] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3123fdbb5c945bd84f97fb4c35c5523. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:44.780900 2203 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:44.781561 6375 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:44.782334 6737 raft_consensus.cc:482] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.782531 6737 raft_consensus.cc:504] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:44.783300 6737 leader_election.cc:288] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:44.784274 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275fc5bbb1494306b1139b7f950ef80f" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:44.784309 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275fc5bbb1494306b1139b7f950ef80f" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:44.784518 4835 raft_consensus.cc:2454] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:44.784569 2203 raft_consensus.cc:2454] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:44.785043 4638 leader_election.cc:302] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:44.785723 6737 raft_consensus.cc:2781] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.785845 6737 raft_consensus.cc:482] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.785955 6737 raft_consensus.cc:3037] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.788877 6737 raft_consensus.cc:504] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:44.792006 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275fc5bbb1494306b1139b7f950ef80f" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:44.792253 4835 raft_consensus.cc:3037] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.792313 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275fc5bbb1494306b1139b7f950ef80f" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:44.792532 2201 raft_consensus.cc:3037] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.795800 2201 raft_consensus.cc:2454] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:44.795774 4835 raft_consensus.cc:2454] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:44.796674 4636 leader_election.cc:302] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:44.797108 6745 raft_consensus.cc:2781] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.798079 6880 tablet_bootstrap.cc:492] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:44.798242 6880 ts_tablet_manager.cc:1367] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.099s user 0.008s sys 0.000s
I20240503 08:12:44.797995 6375 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:44.799111 6880 raft_consensus.cc:348] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.802000 6880 raft_consensus.cc:374] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:44.802408 6880 raft_consensus.cc:729] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:44.802906 1813 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:44.803385 6880 consensus_queue.cc:260] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:44.808707 6880 ts_tablet_manager.cc:1398] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240503 08:12:44.809455 1665 catalog_manager.cc:5484] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:44.811023 6745 raft_consensus.cc:686] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:44.814510 6745 consensus_queue.cc:237] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:44.819061 4703 raft_consensus.cc:1203] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:44.816332 6737 leader_election.cc:288] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:44.825639 6900 ts_tablet_manager.cc:902] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:44.836580 2204 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:44.850837 6867 tablet_copy_client.cc:308] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:41181
I20240503 08:12:44.851972 1833 tablet_copy_service.cc:140] P b8f581390da449ccb9e299011296405e: Received BeginTabletCopySession request for tablet 9e371231b34f45dfac455bbbbe156263 from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:43085)
I20240503 08:12:44.852196 1833 tablet_copy_service.cc:161] P b8f581390da449ccb9e299011296405e: Beginning new tablet copy session on tablet 9e371231b34f45dfac455bbbbe156263 from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:43085: session id = 5f95b05d111e4090bbc59dec394ed0ab-9e371231b34f45dfac455bbbbe156263
I20240503 08:12:44.849509 1664 catalog_manager.cc:5484] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:44.868592 2074 consensus_queue.cc:237] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:44.875175 6838 ts_tablet_manager.cc:902] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:44.875301 1833 tablet_copy_source_session.cc:205] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:44.881841 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e371231b34f45dfac455bbbbe156263. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:12:44.892207 2007 consensus_peers.cc:485] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9cb8a87410b047548efdd973bbc86714. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:44.893666 1943 raft_consensus.cc:1261] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:44.895241 2204 raft_consensus.cc:1261] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:44.897266 6741 consensus_queue.cc:1035] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:44.899474 6741 consensus_queue.cc:1035] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:44.909938 6913 ts_tablet_manager.cc:902] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:44.911339 6741 raft_consensus.cc:2932] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:44.912462 6838 tablet_copy_client.cc:308] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:12:44.913614 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet dab60fefb4704ef19490fb3c76bf47ab from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:56537)
I20240503 08:12:44.913843 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet dab60fefb4704ef19490fb3c76bf47ab from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:56537: session id = 31f41542452b4b1eb994ff4223d039fc-dab60fefb4704ef19490fb3c76bf47ab
I20240503 08:12:44.916064 2204 raft_consensus.cc:2932] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:44.922562 1944 raft_consensus.cc:2932] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:44.923660 1664 catalog_manager.cc:5484] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:44.926283 4988 tablet_copy_source_session.cc:205] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:44.928454 6838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab60fefb4704ef19490fb3c76bf47ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.939015 6838 tablet_copy_client.cc:783] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:44.939504 6838 tablet_copy_client.cc:647] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:44.942345 6838 tablet_copy_client.cc:523] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:44.948429 6745 raft_consensus.cc:482] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:44.948643 6745 raft_consensus.cc:504] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:44.948839 6838 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:44.949496 6745 leader_election.cc:288] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:44.949834 6913 tablet_copy_client.cc:308] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:12:44.950790 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c97a95047e41f9ac878d4755403e98" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:44.951092 2204 raft_consensus.cc:2454] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:44.951705 4636 leader_election.cc:302] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:44.952956 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c97a95047e41f9ac878d4755403e98" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:44.953596 4965 raft_consensus.cc:2454] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:44.954413 6745 raft_consensus.cc:2781] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:44.954524 6745 raft_consensus.cc:482] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:44.954625 6745 raft_consensus.cc:3037] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.958503 6745 raft_consensus.cc:504] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:44.960319 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c97a95047e41f9ac878d4755403e98" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:44.960618 2204 raft_consensus.cc:3037] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.964838 2204 raft_consensus.cc:2454] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:44.965716 4636 leader_election.cc:302] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:44.966498 6897 raft_consensus.cc:2781] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:44.966660 6897 raft_consensus.cc:686] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:44.967051 6897 consensus_queue.cc:237] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:44.968326 6900 tablet_copy_client.cc:308] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:44.969218 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet fa111e798e3f46d2a5bc1eebfe28f8bf from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:43591)
I20240503 08:12:44.969466 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet fa111e798e3f46d2a5bc1eebfe28f8bf from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:43591: session id = 00b61625d8a7460892c118a2570afda5-fa111e798e3f46d2a5bc1eebfe28f8bf
I20240503 08:12:44.969569 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c97a95047e41f9ac878d4755403e98" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:44.969799 4965 raft_consensus.cc:3037] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:44.971028 6930 ts_tablet_manager.cc:902] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:44.972981 1964 tablet_copy_source_session.cc:205] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:44.975240 6867 tablet_copy_client.cc:783] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:44.977523 6867 tablet_copy_client.cc:647] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:44.977820 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa111e798e3f46d2a5bc1eebfe28f8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:44.978289 4965 raft_consensus.cc:2454] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:44.982440 6867 tablet_copy_client.cc:523] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:44.984547 6745 leader_election.cc:288] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:44.997071 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet d6658d7bcd634abd808637e474018965 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a ({username='slave'} at 127.27.157.194:55239)
I20240503 08:12:44.997326 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet d6658d7bcd634abd808637e474018965 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a at {username='slave'} at 127.27.157.194:55239: session id = 315f3f0bcbf447a4b93b1aafcfbf0f5a-d6658d7bcd634abd808637e474018965
I20240503 08:12:44.997593 6900 tablet_copy_client.cc:783] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:44.997920 6900 tablet_copy_client.cc:647] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:45.000380 6867 tablet_bootstrap.cc:492] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:45.000856 6900 tablet_copy_client.cc:523] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:45.002157 4855 tablet_copy_source_session.cc:205] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:45.004796 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6658d7bcd634abd808637e474018965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.009850 6900 tablet_bootstrap.cc:492] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:45.016288 6913 tablet_copy_client.cc:783] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:45.016752 6913 tablet_copy_client.cc:647] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:45.019492 1664 catalog_manager.cc:5484] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:45.020402 6838 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:45.020957 6838 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:12:45.021324 6838 ts_tablet_manager.cc:1367] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.009s
I20240503 08:12:45.022610 6838 raft_consensus.cc:348] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:45.023811 6913 tablet_copy_client.cc:523] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:45.023015 6838 raft_consensus.cc:729] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:12:45.025949 6838 consensus_queue.cc:260] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:45.026926 6838 ts_tablet_manager.cc:1398] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:45.028129 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-dab60fefb4704ef19490fb3c76bf47ab received from {username='slave'} at 127.27.157.198:56537
I20240503 08:12:45.028354 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-dab60fefb4704ef19490fb3c76bf47ab on tablet dab60fefb4704ef19490fb3c76bf47ab with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:45.028669 6913 tablet_bootstrap.cc:492] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:45.051972 6913 tablet_bootstrap.cc:492] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:45.052685 6913 tablet_bootstrap.cc:492] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap complete.
I20240503 08:12:45.052999 6913 ts_tablet_manager.cc:1367] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.006s
I20240503 08:12:45.058594 6930 tablet_copy_client.cc:308] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:45.059711 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet fba1e1952da44c0e956be1ef7f592f08 from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:32893)
I20240503 08:12:45.059916 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet fba1e1952da44c0e956be1ef7f592f08 from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:32893: session id = 5f95b05d111e4090bbc59dec394ed0ab-fba1e1952da44c0e956be1ef7f592f08
I20240503 08:12:45.059760 6913 raft_consensus.cc:348] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:45.060098 6913 raft_consensus.cc:729] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: LEARNER
I20240503 08:12:45.060384 6913 consensus_queue.cc:260] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:45.061949 6913 ts_tablet_manager.cc:1398] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20240503 08:12:45.063098 1964 tablet_copy_source_session.cc:205] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:45.064262 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-d6658d7bcd634abd808637e474018965 received from {username='slave'} at 127.27.157.194:55239
I20240503 08:12:45.064678 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-d6658d7bcd634abd808637e474018965 on tablet d6658d7bcd634abd808637e474018965 with peer 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:12:45.067688 6930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba1e1952da44c0e956be1ef7f592f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.070261 6910 raft_consensus.cc:482] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:45.070472 6910 raft_consensus.cc:504] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:45.071216 6910 leader_election.cc:288] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:45.072557 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6427045c1be84157bfbc066642ff6505" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:45.072872 2074 raft_consensus.cc:2454] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:45.073505 4768 leader_election.cc:302] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:45.074337 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6427045c1be84157bfbc066642ff6505" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:45.074571 4568 raft_consensus.cc:2454] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:45.075466 6910 raft_consensus.cc:2781] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:45.075592 6910 raft_consensus.cc:482] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:45.075671 6910 raft_consensus.cc:3037] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.081457 6910 raft_consensus.cc:504] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:45.082816 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6427045c1be84157bfbc066642ff6505" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:45.083057 2074 raft_consensus.cc:3037] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.086144 2074 raft_consensus.cc:2454] T 6427045c1be84157bfbc066642ff6505 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:45.089908 6930 tablet_copy_client.cc:783] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:45.090344 6930 tablet_copy_client.cc:647] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:45.090350 4768 leader_election.cc:302] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:45.090718 6820 raft_consensus.cc:2781] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:45.092443 6930 tablet_copy_client.cc:523] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:45.095122 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6427045c1be84157bfbc066642ff6505" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:45.095372 4568 raft_consensus.cc:3037] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.098351 4568 raft_consensus.cc:2454] T 6427045c1be84157bfbc066642ff6505 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:45.099162 6820 raft_consensus.cc:686] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:45.099606 6820 consensus_queue.cc:237] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:45.102350 6910 leader_election.cc:288] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:45.113858 6930 tablet_bootstrap.cc:492] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:45.123696 1664 catalog_manager.cc:5484] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:45.152495 1664 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:45.154755 6867 tablet_bootstrap.cc:492] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:45.155470 6867 tablet_bootstrap.cc:492] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:45.155906 6867 ts_tablet_manager.cc:1367] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.156s user 0.044s sys 0.014s
I20240503 08:12:45.156906 6867 raft_consensus.cc:348] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:45.157294 6867 raft_consensus.cc:729] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:45.157584 6867 consensus_queue.cc:260] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:45.164057 6867 ts_tablet_manager.cc:1398] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:12:45.167793 2203 raft_consensus.cc:1261] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:45.167955 4965 raft_consensus.cc:1261] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:45.168589 6745 consensus_queue.cc:1035] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.169245 1833 tablet_copy_service.cc:342] P b8f581390da449ccb9e299011296405e: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-9e371231b34f45dfac455bbbbe156263 received from {username='slave'} at 127.27.157.197:43085
I20240503 08:12:45.169432 1833 tablet_copy_service.cc:433] P b8f581390da449ccb9e299011296405e: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-9e371231b34f45dfac455bbbbe156263 on tablet 9e371231b34f45dfac455bbbbe156263 with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:45.175827 6745 consensus_queue.cc:1035] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.185562 4568 raft_consensus.cc:1203] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:45.211731 4966 consensus_queue.cc:237] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:45.214812 1812 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:45.216125 6593 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:45.220014 6808 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:45.220753 1812 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:45.229543 1672 catalog_manager.cc:5484] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 5 to 7, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:45.230284 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 733bed28775e4779b74e92d8a395234f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:45.232441 6900 tablet_bootstrap.cc:492] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:45.232985 6900 tablet_bootstrap.cc:492] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:12:45.235388 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 733bed28775e4779b74e92d8a395234f with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:45.238754 6896 consensus_queue.cc:1035] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.248901 6586 consensus_queue.cc:1035] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.267786 6964 tablet_replica.cc:330] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:45.268098 6964 raft_consensus.cc:2229] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.268327 6964 raft_consensus.cc:2258] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.270165 6737 consensus_queue.cc:1035] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.275142 6964 ts_tablet_manager.cc:1875] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:45.281750 6900 ts_tablet_manager.cc:1367] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.270s user 0.045s sys 0.017s
I20240503 08:12:45.283380 6900 raft_consensus.cc:348] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:45.285313 6586 consensus_queue.cc:1035] T 09ae3cc0ce534128b49f2da744d23144 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.289940 6900 raft_consensus.cc:729] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:12:45.290328 6900 consensus_queue.cc:260] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:45.291477 6900 ts_tablet_manager.cc:1398] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:12:45.293969 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-fa111e798e3f46d2a5bc1eebfe28f8bf received from {username='slave'} at 127.27.157.200:43591
I20240503 08:12:45.294170 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 00b61625d8a7460892c118a2570afda5-fa111e798e3f46d2a5bc1eebfe28f8bf on tablet fa111e798e3f46d2a5bc1eebfe28f8bf with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:12:45.294740 6962 tablet.cc:1722] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:45.295748 6960 tablet.cc:1722] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:45.296476 6960 tablet.cc:1620] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:45.310139 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=1aeae3adb01442938d48df7fa54e626d] alter complete (version 1)
I20240503 08:12:45.318776 1672 catalog_manager.cc:5576] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183) does not have the latest schema for tablet 07c97a95047e41f9ac878d4755403e98 (table rebalancer_dynamic_table [id=1aeae3adb01442938d48df7fa54e626d]). Expected version 1 got 0
I20240503 08:12:45.321548 2204 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:45.324512 6961 tablet.cc:1722] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:45.325726 4703 raft_consensus.cc:1203] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:45.327690 6593 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:45.328233 6961 tablet.cc:1620] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: MemRowSet was empty: no flush needed.
I20240503 08:12:45.335165 1672 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 07c97a95047e41f9ac878d4755403e98 (table rebalancer_dynamic_table [id=1aeae3adb01442938d48df7fa54e626d]). Expected version 1 got 0
I20240503 08:12:45.335501 6962 tablet.cc:1620] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:45.338285 1664 catalog_manager.cc:5576] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183) does not have the latest schema for tablet 07c97a95047e41f9ac878d4755403e98 (table rebalancer_dynamic_table [id=1aeae3adb01442938d48df7fa54e626d]). Expected version 1 got 0
I20240503 08:12:45.353672 6964 ts_tablet_manager.cc:1888] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:12:45.356583 6964 log.cc:1196] T 733bed28775e4779b74e92d8a395234f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/733bed28775e4779b74e92d8a395234f
I20240503 08:12:45.363750 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 733bed28775e4779b74e92d8a395234f (table rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]) successfully deleted
I20240503 08:12:45.371436 2203 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:45.373771 6900 ts_tablet_manager.cc:902] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:45.377041 6930 tablet_bootstrap.cc:492] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:45.386992 6749 raft_consensus.cc:1050] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:12:45.388121 6749 consensus_queue.cc:237] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:45.392112 6900 tablet_copy_client.cc:308] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:45.393280 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet f3123fdbb5c945bd84f97fb4c35c5523 from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:43591)
I20240503 08:12:45.393498 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet f3123fdbb5c945bd84f97fb4c35c5523 from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:43591: session id = 00b61625d8a7460892c118a2570afda5-f3123fdbb5c945bd84f97fb4c35c5523
I20240503 08:12:45.396883 1964 tablet_copy_source_session.cc:205] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:45.403898 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3123fdbb5c945bd84f97fb4c35c5523. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.403883 6930 tablet_bootstrap.cc:492] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:45.404709 6930 ts_tablet_manager.cc:1367] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.291s user 0.063s sys 0.005s
I20240503 08:12:45.405616 6930 raft_consensus.cc:348] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:45.405908 6930 raft_consensus.cc:729] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:45.406136 6930 consensus_queue.cc:260] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:45.406836 6930 ts_tablet_manager.cc:1398] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240503 08:12:45.410228 4502 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.200:45283, user_credentials={real_user=slave}} blocked reactor thread for 54877.3us
I20240503 08:12:45.414382 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-fba1e1952da44c0e956be1ef7f592f08 received from {username='slave'} at 127.27.157.197:32893
I20240503 08:12:45.414561 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-fba1e1952da44c0e956be1ef7f592f08 on tablet fba1e1952da44c0e956be1ef7f592f08 with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:45.418180 4703 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:45.419204 6735 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:45.423548 6900 tablet_copy_client.cc:783] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:45.423857 6900 tablet_copy_client.cc:647] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:45.424074 1671 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:45.433578 6900 tablet_copy_client.cc:523] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:45.438915 6900 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:45.440258 2204 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:45.443656 6749 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:45.443935 2074 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:45.444977 6624 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:45.446170 6960 tablet.cc:1722] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:45.446724 6960 tablet.cc:1620] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:45.447060 4966 raft_consensus.cc:1203] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:45.452526 6961 tablet.cc:1722] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:45.453222 6961 tablet.cc:1620] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: MemRowSet was empty: no flush needed.
I20240503 08:12:45.460459 6962 tablet.cc:1722] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:45.461167 6962 tablet.cc:1620] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:45.471474 1944 raft_consensus.cc:1203] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:45.481070 1671 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=1aeae3adb01442938d48df7fa54e626d] alter complete (version 2)
I20240503 08:12:45.491757 6930 ts_tablet_manager.cc:902] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:45.511986 6910 consensus_queue.cc:1035] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.515012 6749 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:45.515944 2204 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:45.526002 2074 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:45.526324 1671 catalog_manager.cc:5484] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:45.536460 1810 consensus_queue.cc:237] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:45.538846 2203 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:45.540617 6749 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:45.543989 6624 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:45.545724 2204 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:45.554235 1664 catalog_manager.cc:5484] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:45.554927 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fede923140fb45a5a4f93c0846b62595 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:45.562469 4703 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:45.567107 6900 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:45.567224 6820 consensus_queue.cc:1035] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:45.567500 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet fede923140fb45a5a4f93c0846b62595 with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:45.570375 6900 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:12:45.570700 6900 ts_tablet_manager.cc:1367] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.132s user 0.060s sys 0.017s
I20240503 08:12:45.571656 6900 raft_consensus.cc:348] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:45.571928 6900 raft_consensus.cc:729] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:12:45.572212 6900 consensus_queue.cc:260] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:45.575062 4703 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:45.576426 6749 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:45.578920 6900 ts_tablet_manager.cc:1398] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 08:12:45.579355 4703 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:45.579741 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-f3123fdbb5c945bd84f97fb4c35c5523 received from {username='slave'} at 127.27.157.200:43591
I20240503 08:12:45.579877 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 00b61625d8a7460892c118a2570afda5-f3123fdbb5c945bd84f97fb4c35c5523 on tablet f3123fdbb5c945bd84f97fb4c35c5523 with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:12:45.589993 1672 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:45.599009 6998 tablet_replica.cc:330] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:45.599439 6998 raft_consensus.cc:2229] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.599812 6998 raft_consensus.cc:2258] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.605154 6960 tablet.cc:1722] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:45.606226 6962 tablet.cc:1722] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:45.612022 6961 tablet.cc:1722] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:45.617221 6998 ts_tablet_manager.cc:1875] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:45.621464 1664 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=1aeae3adb01442938d48df7fa54e626d] alter complete (version 3)
I20240503 08:12:45.643121 6998 ts_tablet_manager.cc:1888] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:45.643467 6998 log.cc:1196] T fede923140fb45a5a4f93c0846b62595 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/fede923140fb45a5a4f93c0846b62595
I20240503 08:12:45.650705 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet fede923140fb45a5a4f93c0846b62595 (table rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]) successfully deleted
I20240503 08:12:45.667786 6930 tablet_copy_client.cc:308] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:45.668741 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 9cb8a87410b047548efdd973bbc86714 from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:56593)
I20240503 08:12:45.668951 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 9cb8a87410b047548efdd973bbc86714 from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:56593: session id = 5f95b05d111e4090bbc59dec394ed0ab-9cb8a87410b047548efdd973bbc86714
I20240503 08:12:45.672345 2093 tablet_copy_source_session.cc:205] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:45.674072 6930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb8a87410b047548efdd973bbc86714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.682729 6930 tablet_copy_client.cc:783] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:45.683130 6930 tablet_copy_client.cc:647] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:45.685802 6930 tablet_copy_client.cc:523] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:45.691112 6930 tablet_bootstrap.cc:492] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:45.699106 1672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:45.707746 6624 raft_consensus.cc:1050] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:45.708629 6624 consensus_queue.cc:237] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:45.712436 2073 raft_consensus.cc:1261] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:45.712776 4567 raft_consensus.cc:1261] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:45.714036 6735 consensus_queue.cc:1035] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:45.716640 2203 raft_consensus.cc:1261] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:45.717334 6735 consensus_queue.cc:1035] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:45.722244 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 51f3edb7a72c4ba083429c150fa73363 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a4d2142b44bc4afa85d24f2a4c14b381]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.722813 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f3edb7a72c4ba083429c150fa73363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.727072 4681 tablet_service.cc:1467] Processing CreateTablet for tablet 51f3edb7a72c4ba083429c150fa73363 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a4d2142b44bc4afa85d24f2a4c14b381]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.727576 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f3edb7a72c4ba083429c150fa73363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.732792 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 51f3edb7a72c4ba083429c150fa73363 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a4d2142b44bc4afa85d24f2a4c14b381]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.733121 4549 tablet_service.cc:1467] Processing CreateTablet for tablet c4177c11467b49d49452ac10bd983627 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a4d2142b44bc4afa85d24f2a4c14b381]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.733255 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f3edb7a72c4ba083429c150fa73363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.733601 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4177c11467b49d49452ac10bd983627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.736558 2183 tablet_service.cc:1467] Processing CreateTablet for tablet c4177c11467b49d49452ac10bd983627 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a4d2142b44bc4afa85d24f2a4c14b381]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.736986 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4177c11467b49d49452ac10bd983627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.739890 7018 tablet_bootstrap.cc:492] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:45.739821 4813 tablet_service.cc:1467] Processing CreateTablet for tablet c4177c11467b49d49452ac10bd983627 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a4d2142b44bc4afa85d24f2a4c14b381]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.740309 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4177c11467b49d49452ac10bd983627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.742817 7018 tablet_bootstrap.cc:654] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.746691 7020 tablet_bootstrap.cc:492] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:45.750391 7020 tablet_bootstrap.cc:654] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.753854 1672 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:45.754079 1672 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:45.754976 7018 tablet_bootstrap.cc:492] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:45.756352 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 4422e4f265f344039734893b759c2050 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a4d2142b44bc4afa85d24f2a4c14b381]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.757009 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4422e4f265f344039734893b759c2050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.757993 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 4422e4f265f344039734893b759c2050 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a4d2142b44bc4afa85d24f2a4c14b381]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.758463 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4422e4f265f344039734893b759c2050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.760941 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 4422e4f265f344039734893b759c2050 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a4d2142b44bc4afa85d24f2a4c14b381]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:45.761180 1672 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 07c97a95047e41f9ac878d4755403e98
I20240503 08:12:45.761531 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4422e4f265f344039734893b759c2050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.762825 4946 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c97a95047e41f9ac878d4755403e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:45.765612 2182 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c97a95047e41f9ac878d4755403e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:45.765913 4681 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c97a95047e41f9ac878d4755403e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:45 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:45.766366 7027 tablet_replica.cc:330] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:45.766697 7027 raft_consensus.cc:2229] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.767094 7027 raft_consensus.cc:2258] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.769170 7027 ts_tablet_manager.cc:1875] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.776383 7027 ts_tablet_manager.cc:1888] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:45.779968 7027 log.cc:1196] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/07c97a95047e41f9ac878d4755403e98
I20240503 08:12:45.780680 7027 ts_tablet_manager.cc:1909] T 07c97a95047e41f9ac878d4755403e98 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:45.781240 7026 tablet_replica.cc:330] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:45.781531 7026 raft_consensus.cc:2229] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:45.781801 7026 raft_consensus.cc:2258] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.779973 7021 tablet_bootstrap.cc:492] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:45.782016 7018 ts_tablet_manager.cc:1367] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.010s
I20240503 08:12:45.782088 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 07c97a95047e41f9ac878d4755403e98 (table rebalancer_dynamic_table_new_name [id=1aeae3adb01442938d48df7fa54e626d]) successfully deleted
I20240503 08:12:45.782688 7028 tablet_replica.cc:330] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:45.782974 7028 raft_consensus.cc:2229] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:45.784497 7021 tablet_bootstrap.cc:654] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.787288 7029 tablet_bootstrap.cc:492] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:45.786903 7018 raft_consensus.cc:348] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.787526 7018 raft_consensus.cc:374] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.787652 7018 raft_consensus.cc:729] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.788111 7028 raft_consensus.cc:2258] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:45.788146 7018 consensus_queue.cc:260] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.789101 7026 ts_tablet_manager.cc:1875] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.790779 6624 consensus_queue.cc:1035] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20240503 08:12:45.791926 6735 raft_consensus.cc:2932] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:45.793202 7021 tablet_bootstrap.cc:492] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:45.793454 7021 ts_tablet_manager.cc:1367] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240503 08:12:45.794423 7021 raft_consensus.cc:348] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.794714 7021 raft_consensus.cc:374] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.794859 7021 raft_consensus.cc:729] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.795099 7018 ts_tablet_manager.cc:1398] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
I20240503 08:12:45.795264 7021 consensus_queue.cc:260] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.791018 7028 ts_tablet_manager.cc:1875] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:45.799446 7021 ts_tablet_manager.cc:1398] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:45.799842 7021 tablet_bootstrap.cc:492] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:45.799628 2203 raft_consensus.cc:2932] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:45.800612 7018 tablet_bootstrap.cc:492] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:45.801927 7021 tablet_bootstrap.cc:654] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.803361 7018 tablet_bootstrap.cc:654] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.805051 7029 tablet_bootstrap.cc:654] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.806416 7023 tablet_bootstrap.cc:492] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:45.808219 7023 tablet_bootstrap.cc:654] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.810947 4567 raft_consensus.cc:2932] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:45.814332 7023 tablet_bootstrap.cc:492] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:45.815279 7023 ts_tablet_manager.cc:1367] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 08:12:45.816483 7023 raft_consensus.cc:348] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.817852 7023 raft_consensus.cc:374] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.818342 4568 raft_consensus.cc:1203] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:45.818603 7023 raft_consensus.cc:729] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.819286 7023 consensus_queue.cc:260] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.820365 7023 ts_tablet_manager.cc:1398] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:45.821235 7021 tablet_bootstrap.cc:492] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:45.821483 7021 ts_tablet_manager.cc:1367] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.001s
I20240503 08:12:45.822602 7021 raft_consensus.cc:348] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:45.822937 7021 raft_consensus.cc:374] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.823100 7021 raft_consensus.cc:729] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.824291 1671 catalog_manager.cc:5484] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:45.823571 7021 consensus_queue.cc:260] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:45.826213 7029 tablet_bootstrap.cc:492] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:45.826339 7021 ts_tablet_manager.cc:1398] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:45.826447 7029 ts_tablet_manager.cc:1367] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.003s
I20240503 08:12:45.826936 7023 tablet_bootstrap.cc:492] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:45.827289 7029 raft_consensus.cc:348] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:45.827633 7029 raft_consensus.cc:374] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.827853 7029 raft_consensus.cc:729] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.828912 7023 tablet_bootstrap.cc:654] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.831605 7029 consensus_queue.cc:260] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:45.832665 7029 ts_tablet_manager.cc:1398] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 08:12:45.838240 7023 tablet_bootstrap.cc:492] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:45.838553 7023 ts_tablet_manager.cc:1367] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240503 08:12:45.839560 7023 raft_consensus.cc:348] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.839885 7023 raft_consensus.cc:374] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.839998 4966 raft_consensus.cc:1203] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:45.840150 7023 raft_consensus.cc:729] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.840469 7023 consensus_queue.cc:260] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.841185 7023 ts_tablet_manager.cc:1398] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:45.841544 7023 tablet_bootstrap.cc:492] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:45.845111 7023 tablet_bootstrap.cc:654] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.853403 7023 tablet_bootstrap.cc:492] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:45.853703 7023 ts_tablet_manager.cc:1367] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 08:12:45.855381 7023 raft_consensus.cc:348] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:45.855732 7023 raft_consensus.cc:374] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.855890 7023 raft_consensus.cc:729] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.856231 7023 consensus_queue.cc:260] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:45.856985 7023 ts_tablet_manager.cc:1398] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:45.862244 7018 tablet_bootstrap.cc:492] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:45.862537 7018 ts_tablet_manager.cc:1367] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.062s user 0.000s sys 0.004s
I20240503 08:12:45.862708 2073 raft_consensus.cc:2932] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:45.864328 7018 raft_consensus.cc:348] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.864702 7018 raft_consensus.cc:374] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.864842 7018 raft_consensus.cc:729] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.865181 7018 consensus_queue.cc:260] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.865643 6820 raft_consensus.cc:1050] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a to VOTER
I20240503 08:12:45.866341 7018 ts_tablet_manager.cc:1398] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:45.866533 6820 consensus_queue.cc:237] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:12:45.873407 6972 raft_consensus.cc:1050] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:12:45.874414 6972 consensus_queue.cc:237] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:45.882293 4703 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:45.883049 6808 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:45.885224 1672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:45.895228 4835 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:45.896171 2074 raft_consensus.cc:1261] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:45.896226 4966 raft_consensus.cc:1261] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:45.896961 6910 consensus_queue.cc:1035] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:45.897636 6910 consensus_queue.cc:1035] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:45.898150 6808 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:45.898690 6930 tablet_bootstrap.cc:492] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:45.899191 6930 tablet_bootstrap.cc:492] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:45.899513 6930 ts_tablet_manager.cc:1367] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.208s user 0.060s sys 0.000s
I20240503 08:12:45.900436 6930 raft_consensus.cc:348] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:45.900699 6930 raft_consensus.cc:729] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:45.900960 6930 consensus_queue.cc:260] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:45.901480 1943 raft_consensus.cc:1261] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:45.901767 6930 ts_tablet_manager.cc:1398] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:45.902559 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-9cb8a87410b047548efdd973bbc86714 received from {username='slave'} at 127.27.157.197:56593
I20240503 08:12:45.902709 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-9cb8a87410b047548efdd973bbc86714 on tablet 9cb8a87410b047548efdd973bbc86714 with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:45.902551 6947 consensus_queue.cc:1035] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:45.909019 4569 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:45.910526 6972 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:45.914649 6647 raft_consensus.cc:482] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:45.914912 6647 raft_consensus.cc:504] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.916047 6647 leader_election.cc:288] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:45.917860 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4177c11467b49d49452ac10bd983627" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:45.918139 4834 raft_consensus.cc:2454] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:45.918910 2140 leader_election.cc:302] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:45.918911 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4177c11467b49d49452ac10bd983627" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:45.919845 6647 raft_consensus.cc:2781] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:45.919941 6647 raft_consensus.cc:482] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:45.920043 6647 raft_consensus.cc:3037] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.921694 6947 raft_consensus.cc:2932] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:45.922480 4567 raft_consensus.cc:2454] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:45.926406 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63]), partition=
I20240503 08:12:45.926893 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575dd6e43e3e41d294b6eb27ef7c68d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.926914 4681 tablet_service.cc:1467] Processing CreateTablet for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63]), partition=
I20240503 08:12:45.927493 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575dd6e43e3e41d294b6eb27ef7c68d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.930881 1668 catalog_manager.cc:5484] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:45.928586 2074 raft_consensus.cc:2932] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:45.933880 7018 tablet_bootstrap.cc:492] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
W20240503 08:12:45.934943 7031 log.cc:924] Time spent T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20240503 08:12:45.935556 7018 tablet_bootstrap.cc:654] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.940263 4966 raft_consensus.cc:2932] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:45.946195 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63]), partition=
I20240503 08:12:45.946539 7026 ts_tablet_manager.cc:1888] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:45.946615 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575dd6e43e3e41d294b6eb27ef7c68d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:45.946715 7026 log.cc:1196] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/07c97a95047e41f9ac878d4755403e98
I20240503 08:12:45.947238 7026 ts_tablet_manager.cc:1909] T 07c97a95047e41f9ac878d4755403e98 P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:45.949573 7020 tablet_bootstrap.cc:492] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:45.949759 7020 ts_tablet_manager.cc:1367] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.203s user 0.004s sys 0.004s
I20240503 08:12:45.950784 7020 raft_consensus.cc:348] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.951035 7020 raft_consensus.cc:374] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.951151 7020 raft_consensus.cc:729] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.951578 7020 consensus_queue.cc:260] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.952199 7020 ts_tablet_manager.cc:1398] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:45.953519 6647 raft_consensus.cc:504] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.961272 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4177c11467b49d49452ac10bd983627" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:45.961557 4567 raft_consensus.cc:3037] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.962386 7028 ts_tablet_manager.cc:1888] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:45.962577 7028 log.cc:1196] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/07c97a95047e41f9ac878d4755403e98
I20240503 08:12:45.963127 7028 ts_tablet_manager.cc:1909] T 07c97a95047e41f9ac878d4755403e98 P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:45.964994 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 07c97a95047e41f9ac878d4755403e98 (table rebalancer_dynamic_table_new_name [id=1aeae3adb01442938d48df7fa54e626d]) successfully deleted
I20240503 08:12:45.965358 4567 raft_consensus.cc:2454] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:45.972561 6647 leader_election.cc:288] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:45.964603 6972 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:45.978173 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4177c11467b49d49452ac10bd983627" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:45.978492 4835 raft_consensus.cc:3037] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:45.979463 4834 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:45.983398 4835 raft_consensus.cc:2454] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:45.985482 2140 leader_election.cc:302] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:45.991760 7051 raft_consensus.cc:2781] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:45.993582 7051 raft_consensus.cc:686] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:45.994457 7018 tablet_bootstrap.cc:492] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:45.995400 7051 consensus_queue.cc:237] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.996155 7023 tablet_bootstrap.cc:492] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:45.997874 7018 ts_tablet_manager.cc:1367] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.064s user 0.000s sys 0.012s
I20240503 08:12:45.997948 7023 tablet_bootstrap.cc:654] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:45.998669 7018 raft_consensus.cc:348] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:45.998855 7018 raft_consensus.cc:374] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:45.998957 7018 raft_consensus.cc:729] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:45.999797 7018 consensus_queue.cc:260] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:46.000455 7018 ts_tablet_manager.cc:1398] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:46.000245 1943 raft_consensus.cc:2932] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:46.004657 4701 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:46.008055 1665 catalog_manager.cc:5484] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 2 to 3, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:46.020020 4568 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:46.028731 7023 tablet_bootstrap.cc:492] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:46.028955 7023 ts_tablet_manager.cc:1367] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.005s
I20240503 08:12:46.029454 4566 raft_consensus.cc:1203] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:46.029737 7023 raft_consensus.cc:348] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:46.029974 7023 raft_consensus.cc:374] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.030053 7023 raft_consensus.cc:729] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.030269 7023 consensus_queue.cc:260] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:46.030894 7023 ts_tablet_manager.cc:1398] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:46.036471 1664 catalog_manager.cc:5484] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:46.039638 6559 raft_consensus.cc:1050] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:12:46.040946 6559 consensus_queue.cc:237] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:46.056946 7020 tablet_bootstrap.cc:492] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:46.057226 4966 consensus_queue.cc:237] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:46.060124 4701 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:46.060400 6947 raft_consensus.cc:482] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:46.060590 6947 raft_consensus.cc:504] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:46.060890 7050 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:46.061472 6947 leader_election.cc:288] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:46.063177 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f3edb7a72c4ba083429c150fa73363" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:46.065316 7050 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:46.066190 4569 raft_consensus.cc:2454] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:46.062786 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 07c97a95047e41f9ac878d4755403e98 (table rebalancer_dynamic_table_new_name [id=1aeae3adb01442938d48df7fa54e626d]) successfully deleted
I20240503 08:12:46.066684 4767 leader_election.cc:302] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:46.067028 6820 raft_consensus.cc:2781] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:46.067739 6820 raft_consensus.cc:482] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:46.067899 6820 raft_consensus.cc:3037] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.069885 7020 tablet_bootstrap.cc:654] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.070628 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f3edb7a72c4ba083429c150fa73363" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:46.070991 4699 raft_consensus.cc:2454] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:46.071779 4699 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:46.073194 6820 raft_consensus.cc:504] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:46.077522 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f3edb7a72c4ba083429c150fa73363" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:46.077826 4701 raft_consensus.cc:3037] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.078984 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dab60fefb4704ef19490fb3c76bf47ab with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:46.080319 4834 consensus_queue.cc:237] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:12:46.082983 4966 raft_consensus.cc:1261] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:46.084338 4701 raft_consensus.cc:2454] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:46.084456 1943 raft_consensus.cc:1261] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:46.084986 4770 leader_election.cc:302] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:46.085652 6910 raft_consensus.cc:2781] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:46.086138 6947 consensus_queue.cc:1035] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:46.087690 6947 consensus_queue.cc:1035] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:46.096899 7020 tablet_bootstrap.cc:492] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:46.094807 1671 catalog_manager.cc:5484] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:46.097250 7020 ts_tablet_manager.cc:1367] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.009s
I20240503 08:12:46.098210 7020 raft_consensus.cc:348] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:46.098568 7020 raft_consensus.cc:374] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.098549 6820 leader_election.cc:288] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:46.098740 7020 raft_consensus.cc:729] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.099105 7020 consensus_queue.cc:260] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:46.100240 7020 ts_tablet_manager.cc:1398] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 08:12:46.103672 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f3edb7a72c4ba083429c150fa73363" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:46.104969 4568 raft_consensus.cc:3037] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.063889 4567 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:46.113514 7048 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:46.116319 4567 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:46.117493 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet dab60fefb4704ef19490fb3c76bf47ab with delete_type TABLET_DATA_TOMBSTONED (TS f24648c92c384f1ba349eee801f985e2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:46.121019 6947 raft_consensus.cc:2932] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:46.126308 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6658d7bcd634abd808637e474018965 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:46.133360 1944 raft_consensus.cc:2932] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:46.154120 1664 catalog_manager.cc:5484] T d6658d7bcd634abd808637e474018965 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:46.156389 6970 raft_consensus.cc:482] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:46.156662 6970 raft_consensus.cc:504] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:46.158031 6970 leader_election.cc:288] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:46.158860 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575dd6e43e3e41d294b6eb27ef7c68d0" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:46.159157 4834 raft_consensus.cc:2454] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:46.159294 4568 raft_consensus.cc:2454] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:46.160326 4504 leader_election.cc:302] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:46.160768 6970 raft_consensus.cc:2781] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:46.160900 6970 raft_consensus.cc:482] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:46.161008 6970 raft_consensus.cc:3037] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.164984 6970 raft_consensus.cc:504] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:46.165277 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet d6658d7bcd634abd808637e474018965 with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:46.166213 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575dd6e43e3e41d294b6eb27ef7c68d0" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:46.166438 4699 raft_consensus.cc:3037] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.170354 4699 raft_consensus.cc:2454] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:46.171012 4504 leader_election.cc:302] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:46.172791 7065 tablet_replica.cc:330] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:46.172753 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575dd6e43e3e41d294b6eb27ef7c68d0" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:46.173764 6970 leader_election.cc:288] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:46.173781 7065 raft_consensus.cc:2229] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:46.174058 7065 raft_consensus.cc:2258] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.174222 4701 raft_consensus.cc:2362] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1. Re-sending same reply.
I20240503 08:12:46.175175 6970 raft_consensus.cc:2781] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:46.175257 7064 tablet_replica.cc:330] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:46.175601 7064 raft_consensus.cc:2229] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:46.175750 6970 raft_consensus.cc:686] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:46.175992 7064 raft_consensus.cc:2258] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.176353 6970 consensus_queue.cc:237] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:46.176550 6910 raft_consensus.cc:686] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:46.176918 6910 consensus_queue.cc:237] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:46.178596 7060 raft_consensus.cc:482] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:46.178727 7065 ts_tablet_manager.cc:1875] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:46.178740 7060 raft_consensus.cc:504] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:46.179610 7060 leader_election.cc:288] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:46.184224 7064 ts_tablet_manager.cc:1875] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:46.184577 4966 raft_consensus.cc:1261] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.185719 1810 raft_consensus.cc:1261] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.187076 2203 raft_consensus.cc:1261] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.188870 6375 consensus_queue.cc:1035] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.189781 6375 consensus_queue.cc:1035] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.190627 6375 consensus_queue.cc:1035] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.194924 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4422e4f265f344039734893b759c2050" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:46.195255 2203 raft_consensus.cc:2454] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:46.196336 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4422e4f265f344039734893b759c2050" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:46.196642 2074 raft_consensus.cc:2454] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:46.197432 4768 leader_election.cc:302] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:46.198801 7061 raft_consensus.cc:2781] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:46.198969 7061 raft_consensus.cc:482] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:46.199139 7061 raft_consensus.cc:3037] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.205221 7064 ts_tablet_manager.cc:1888] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:46.209127 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575dd6e43e3e41d294b6eb27ef7c68d0" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:46.209491 4834 raft_consensus.cc:3037] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.209589 7061 raft_consensus.cc:504] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:46.211512 4967 raft_consensus.cc:2932] T d6658d7bcd634abd808637e474018965 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:46.212628 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4422e4f265f344039734893b759c2050" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:46.212939 2203 raft_consensus.cc:3037] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.215831 2203 raft_consensus.cc:2454] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:46.217016 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4422e4f265f344039734893b759c2050" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:46.217128 4834 raft_consensus.cc:2454] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:46.217250 2074 raft_consensus.cc:3037] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:46.217468 4768 leader_election.cc:302] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:46.218062 1664 catalog_manager.cc:5484] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:46.218472 7061 leader_election.cc:288] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:46.218680 7062 raft_consensus.cc:2781] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:46.218825 7062 raft_consensus.cc:686] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:46.219183 7062 consensus_queue.cc:237] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:46.221184 7064 log.cc:1196] T dab60fefb4704ef19490fb3c76bf47ab P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/dab60fefb4704ef19490fb3c76bf47ab
I20240503 08:12:46.223778 2074 raft_consensus.cc:2454] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:46.226266 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet dab60fefb4704ef19490fb3c76bf47ab (table rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]) successfully deleted
I20240503 08:12:46.228605 7013 raft_consensus.cc:2932] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:46.238669 1810 raft_consensus.cc:2932] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:46.249434 7065 ts_tablet_manager.cc:1888] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:46.249605 7065 log.cc:1196] T d6658d7bcd634abd808637e474018965 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/d6658d7bcd634abd808637e474018965
I20240503 08:12:46.254766 1664 catalog_manager.cc:5484] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:46.257283 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet d6658d7bcd634abd808637e474018965 (table rebalancer_extra_table_6 [id=6b5d1749a8944d598384a9bb153c5fc1]) successfully deleted
I20240503 08:12:46.252414 4967 raft_consensus.cc:2932] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:46.268303 1813 consensus_queue.cc:237] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:46.269102 1671 catalog_manager.cc:5484] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:46.272630 2073 raft_consensus.cc:1261] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:46.273733 7015 consensus_queue.cc:1035] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:46.277652 1944 consensus_queue.cc:237] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:46.279907 7015 raft_consensus.cc:2932] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.284422 2073 raft_consensus.cc:2932] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.293913 4569 raft_consensus.cc:1261] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:46.294131 4967 raft_consensus.cc:1261] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:46.294790 1813 raft_consensus.cc:1261] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:46.298599 6749 consensus_queue.cc:1035] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:46.300781 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e371231b34f45dfac455bbbbe156263 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:46.300117 7013 consensus_queue.cc:1035] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:46.302755 7013 consensus_queue.cc:1035] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:46.306249 4569 raft_consensus.cc:2932] T 9e371231b34f45dfac455bbbbe156263 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.308403 1672 catalog_manager.cc:5484] T 9e371231b34f45dfac455bbbbe156263 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.315007 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e371231b34f45dfac455bbbbe156263 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:46.316097 7079 tablet_replica.cc:330] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:46.316090 6375 raft_consensus.cc:2932] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:46.318754 7079 raft_consensus.cc:2229] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:46.321810 1810 raft_consensus.cc:2932] T fa111e798e3f46d2a5bc1eebfe28f8bf P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:46.323892 7079 raft_consensus.cc:2258] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.325434 4966 raft_consensus.cc:2932] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:46.328184 1664 catalog_manager.cc:5484] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:46.333709 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fa111e798e3f46d2a5bc1eebfe28f8bf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:46.338511 1672 catalog_manager.cc:5484] T fa111e798e3f46d2a5bc1eebfe28f8bf P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:46.342156 7079 ts_tablet_manager.cc:1875] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:46.351822 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet fa111e798e3f46d2a5bc1eebfe28f8bf with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:46.362807 1672 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:46.377164 7079 ts_tablet_manager.cc:1888] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:46.378023 7079 log.cc:1196] T 9e371231b34f45dfac455bbbbe156263 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/9e371231b34f45dfac455bbbbe156263
I20240503 08:12:46.378501 4834 raft_consensus.cc:1261] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:46.378507 4701 raft_consensus.cc:1261] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:46.379447 7066 consensus_queue.cc:1035] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.379699 7079 tablet_replica.cc:330] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:46.379772 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 9e371231b34f45dfac455bbbbe156263 (table rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]) successfully deleted
I20240503 08:12:46.379813 7074 consensus_queue.cc:1035] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.380106 7079 raft_consensus.cc:2229] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:46.380300 7079 pending_rounds.cc:62] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5: Trying to abort 1 pending ops.
I20240503 08:12:46.382031 7079 pending_rounds.cc:68] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023509364900429824 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "fa111e798e3f46d2a5bc1eebfe28f8bf" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } } }
I20240503 08:12:46.382715 7079 raft_consensus.cc:2894] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 08:12:46.382905 7079 raft_consensus.cc:2866] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:12:46.383950 7079 raft_consensus.cc:2258] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.386399 7079 ts_tablet_manager.cc:1875] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:46.401607 7079 ts_tablet_manager.cc:1888] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:46.404574 7079 log.cc:1196] T fa111e798e3f46d2a5bc1eebfe28f8bf P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/fa111e798e3f46d2a5bc1eebfe28f8bf
I20240503 08:12:46.405622 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet fa111e798e3f46d2a5bc1eebfe28f8bf (table rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]) successfully deleted
I20240503 08:12:46.412325 1672 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (table rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63]). Expected version 1 got 0
I20240503 08:12:46.413769 1667 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (table rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63]). Expected version 1 got 0
I20240503 08:12:46.416226 7091 tablet.cc:1722] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:46.417284 7091 tablet.cc:1620] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:46.426270 1667 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63] alter complete (version 1)
I20240503 08:12:46.432579 6647 consensus_queue.cc:1035] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.437685 1672 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (table rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63]). Expected version 1 got 0
I20240503 08:12:46.439632 6559 raft_consensus.cc:1050] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:46.440552 6559 consensus_queue.cc:237] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:46.440955 6647 consensus_queue.cc:1035] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.449939 1667 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (table rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63]). Expected version 1 got 0
I20240503 08:12:46.454465 7090 tablet.cc:1722] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:46.455271 7090 tablet.cc:1620] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:46.461632 7013 raft_consensus.cc:1050] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:12:46.462777 7013 consensus_queue.cc:237] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:46.470925 6741 raft_consensus.cc:1050] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:46.472597 6741 consensus_queue.cc:237] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:46.474188 4568 raft_consensus.cc:1261] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.474879 6375 consensus_queue.cc:1035] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.485805 2073 raft_consensus.cc:1261] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.486280 2204 raft_consensus.cc:1261] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.486692 6375 consensus_queue.cc:1035] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.487493 6375 consensus_queue.cc:1035] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.499615 1810 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.501529 7102 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.509435 7093 tablet.cc:1722] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:46.512225 4568 raft_consensus.cc:1261] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.513212 2204 raft_consensus.cc:1261] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.513764 7055 consensus_queue.cc:1035] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.516919 7055 consensus_queue.cc:1035] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.519869 7093 tablet.cc:1620] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:46.519837 1944 raft_consensus.cc:1261] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.522266 1667 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (table rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63]). Expected version 1 got 0
I20240503 08:12:46.526731 6609 consensus_queue.cc:1035] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.543761 1667 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:46.569049 1667 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (table rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63]). Expected version 2 got 1
W20240503 08:12:46.572302 1878 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.196:42183, user_credentials={real_user=slave}} blocked reactor thread for 85520.8us
I20240503 08:12:46.573525 4966 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.575815 6375 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.577579 7091 tablet.cc:1722] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:46.578176 7091 tablet.cc:1620] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:46.584091 2203 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:46.587479 1667 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (table rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63]). Expected version 2 got 1
I20240503 08:12:46.588595 7093 tablet.cc:1722] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:46.589296 7093 tablet.cc:1620] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:46.590665 7090 tablet.cc:1722] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:46.591449 7090 tablet.cc:1620] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:46.597759 6375 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.604494 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=90d6d1c2e2394f03b8238399f8666a63] alter complete (version 2)
I20240503 08:12:46.604465 6741 raft_consensus.cc:2932] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.606503 7062 consensus_queue.cc:1035] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.610404 4569 raft_consensus.cc:2932] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.613162 7101 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:46.619520 1810 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:46.623013 1672 catalog_manager.cc:5484] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:46.623744 2204 raft_consensus.cc:2932] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.629863 1664 catalog_manager.cc:5484] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:46.630762 7013 raft_consensus.cc:2932] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.631875 4966 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:46.636253 7061 consensus_queue.cc:1035] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.638542 4569 raft_consensus.cc:2932] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.641719 2202 raft_consensus.cc:2932] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.649572 2073 consensus_queue.cc:237] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:46.657107 4566 raft_consensus.cc:1261] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:46.657694 1943 raft_consensus.cc:1261] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:46.658361 7110 consensus_queue.cc:1035] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:46.659605 2072 raft_consensus.cc:2932] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.660310 6741 consensus_queue.cc:1035] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:46.662236 1943 raft_consensus.cc:2932] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.668905 4566 raft_consensus.cc:2932] T 9cb8a87410b047548efdd973bbc86714 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.669837 1667 catalog_manager.cc:5484] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:46.671960 6741 raft_consensus.cc:2932] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.676337 1942 raft_consensus.cc:2932] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.689363 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9cb8a87410b047548efdd973bbc86714 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:46.693132 1942 consensus_queue.cc:237] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:46.695230 1667 catalog_manager.cc:5484] T 9cb8a87410b047548efdd973bbc86714 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.696202 4566 raft_consensus.cc:1261] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:46.696892 6375 consensus_queue.cc:1035] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:46.698839 2072 raft_consensus.cc:1261] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:46.699556 6375 consensus_queue.cc:1035] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:46.700673 6375 raft_consensus.cc:2932] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.705332 4568 raft_consensus.cc:2932] T fba1e1952da44c0e956be1ef7f592f08 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
W20240503 08:12:46.711930 2140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.198:46559, user_credentials={real_user=slave}} blocked reactor thread for 76755.5us
I20240503 08:12:46.713171 2072 raft_consensus.cc:2932] T fba1e1952da44c0e956be1ef7f592f08 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:46.716264 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fba1e1952da44c0e956be1ef7f592f08 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:46.720372 1672 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:46.725445 1668 catalog_manager.cc:5484] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:46.744446 1667 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (table rebalancer_dynamic_table_new_name [id=90d6d1c2e2394f03b8238399f8666a63]). Expected version 3 got 2
I20240503 08:12:46.749732 7091 tablet.cc:1722] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:46.750660 7093 tablet.cc:1722] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:46.756525 7090 tablet.cc:1722] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:46.764551 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=90d6d1c2e2394f03b8238399f8666a63] alter complete (version 3)
I20240503 08:12:46.767822 2202 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:46.782577 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb8a87410b047548efdd973bbc86714 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:46.783004 7079 tablet_replica.cc:330] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:46.783280 7079 raft_consensus.cc:2229] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:46.783581 7079 raft_consensus.cc:2258] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.783030 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet fba1e1952da44c0e956be1ef7f592f08 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:46.785609 7079 ts_tablet_manager.cc:1875] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:46.786499 6910 consensus_queue.cc:1035] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.798727 7079 ts_tablet_manager.cc:1888] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:46.800293 7079 log.cc:1196] T 9cb8a87410b047548efdd973bbc86714 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/9cb8a87410b047548efdd973bbc86714
I20240503 08:12:46.801597 7079 tablet_replica.cc:330] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:46.801632 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 9cb8a87410b047548efdd973bbc86714 (table rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]) successfully deleted
I20240503 08:12:46.802821 7079 raft_consensus.cc:2229] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:46.803601 7079 raft_consensus.cc:2258] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.808450 7079 ts_tablet_manager.cc:1875] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:46.819998 7079 ts_tablet_manager.cc:1888] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:46.820309 7079 log.cc:1196] T fba1e1952da44c0e956be1ef7f592f08 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/fba1e1952da44c0e956be1ef7f592f08
I20240503 08:12:46.821210 7061 consensus_queue.cc:1035] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:46.823740 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet fba1e1952da44c0e956be1ef7f592f08 (table rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]) successfully deleted
I20240503 08:12:46.836771 1944 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f3123fdbb5c945bd84f97fb4c35c5523"
dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57144
I20240503 08:12:46.837011 1944 raft_consensus.cc:595] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:46.869501 1671 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:46.882144 4681 tablet_service.cc:1467] Processing CreateTablet for tablet fe20c189635648f5b78905e8b9643bcd (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b6256afd3f1a487c896a8ef180a92b10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.882748 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe20c189635648f5b78905e8b9643bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.882843 1924 tablet_service.cc:1467] Processing CreateTablet for tablet fe20c189635648f5b78905e8b9643bcd (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b6256afd3f1a487c896a8ef180a92b10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.883313 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe20c189635648f5b78905e8b9643bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.883291 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 6862b07fa36f4a458913803367c3f23e (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b6256afd3f1a487c896a8ef180a92b10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.883806 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6862b07fa36f4a458913803367c3f23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.885460 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 40476d1762154f6d9ce4855872492573 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b6256afd3f1a487c896a8ef180a92b10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.885901 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40476d1762154f6d9ce4855872492573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.887814 2184 tablet_service.cc:1467] Processing CreateTablet for tablet fe20c189635648f5b78905e8b9643bcd (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b6256afd3f1a487c896a8ef180a92b10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.888253 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe20c189635648f5b78905e8b9643bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.891670 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 40476d1762154f6d9ce4855872492573 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b6256afd3f1a487c896a8ef180a92b10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.891952 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 6862b07fa36f4a458913803367c3f23e (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b6256afd3f1a487c896a8ef180a92b10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.892161 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40476d1762154f6d9ce4855872492573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.892172 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 6862b07fa36f4a458913803367c3f23e (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b6256afd3f1a487c896a8ef180a92b10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.892411 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6862b07fa36f4a458913803367c3f23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.892582 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6862b07fa36f4a458913803367c3f23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.895789 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 40476d1762154f6d9ce4855872492573 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=b6256afd3f1a487c896a8ef180a92b10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:46.896267 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40476d1762154f6d9ce4855872492573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:46.900528 1668 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:46.900774 1668 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:46.906971 1668 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 575dd6e43e3e41d294b6eb27ef7c68d0
I20240503 08:12:46.908386 4549 tablet_service.cc:1514] Processing DeleteTablet for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:46 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:46.908914 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:46 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:46.918810 2202 consensus_queue.cc:237] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:46.923520 7156 tablet_bootstrap.cc:492] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:46.924089 7158 tablet_replica.cc:330] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:46.924651 7158 raft_consensus.cc:2229] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:46.926657 7156 tablet_bootstrap.cc:654] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.926993 7158 raft_consensus.cc:2258] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.929306 7158 ts_tablet_manager.cc:1875] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:46.931492 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 575dd6e43e3e41d294b6eb27ef7c68d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:46 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:46.940135 7156 tablet_bootstrap.cc:492] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:46.940337 7156 ts_tablet_manager.cc:1367] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 08:12:46.941241 7156 raft_consensus.cc:348] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:46.941526 7156 raft_consensus.cc:374] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.941630 7156 raft_consensus.cc:729] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.946777 7159 tablet_bootstrap.cc:492] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:46.947974 7157 tablet_replica.cc:330] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:46.948698 7154 tablet_bootstrap.cc:492] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:46.950346 7159 tablet_bootstrap.cc:654] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.950855 7164 tablet_replica.cc:330] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:46.952628 7161 tablet_bootstrap.cc:492] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:46.949357 7155 tablet_bootstrap.cc:492] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:46.952685 7157 raft_consensus.cc:2229] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:46.953083 7164 raft_consensus.cc:2229] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:46.953701 7163 tablet_bootstrap.cc:492] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:46.954258 7157 raft_consensus.cc:2258] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.954459 7161 tablet_bootstrap.cc:654] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.954715 7164 raft_consensus.cc:2258] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:46.949585 7156 consensus_queue.cc:260] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:46.955307 7156 ts_tablet_manager.cc:1398] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.015s user 0.001s sys 0.002s
I20240503 08:12:46.954571 7155 tablet_bootstrap.cc:654] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.956815 7156 tablet_bootstrap.cc:492] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:46.957937 7164 ts_tablet_manager.cc:1875] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:46.958710 7156 tablet_bootstrap.cc:654] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.959720 7158 ts_tablet_manager.cc:1888] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
W20240503 08:12:46.960319 2140 consensus_peers.cc:485] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc342fed6c6343a080ae0ff2c53cb140. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:46.960551 7158 log.cc:1196] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/575dd6e43e3e41d294b6eb27ef7c68d0
I20240503 08:12:46.961205 7158 ts_tablet_manager.cc:1909] T 575dd6e43e3e41d294b6eb27ef7c68d0 P f24648c92c384f1ba349eee801f985e2: Deleting consensus metadata
I20240503 08:12:46.961364 7157 ts_tablet_manager.cc:1875] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:46.959148 7154 tablet_bootstrap.cc:654] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.963053 1810 raft_consensus.cc:1261] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:46.963948 7056 consensus_queue.cc:1035] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:46.965201 1944 raft_consensus.cc:1261] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:46.967159 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (table rebalancer_dynamic_table_new_name [id=90d6d1c2e2394f03b8238399f8666a63]) successfully deleted
I20240503 08:12:46.967446 7056 consensus_queue.cc:1035] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:46.974035 7154 tablet_bootstrap.cc:492] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:46.974304 7154 ts_tablet_manager.cc:1367] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.000s
I20240503 08:12:46.975351 7154 raft_consensus.cc:348] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:46.976514 7154 raft_consensus.cc:374] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.976686 7154 raft_consensus.cc:729] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.977011 7154 consensus_queue.cc:260] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:46.978869 7154 ts_tablet_manager.cc:1398] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:46.983636 7154 tablet_bootstrap.cc:492] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:46.989214 7156 tablet_bootstrap.cc:492] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:46.989581 7156 ts_tablet_manager.cc:1367] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.003s
I20240503 08:12:46.990844 7156 raft_consensus.cc:348] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:46.991187 7156 raft_consensus.cc:374] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:46.991309 7156 raft_consensus.cc:729] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:46.991674 7156 consensus_queue.cc:260] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:46.993917 7156 ts_tablet_manager.cc:1398] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:46.996449 7154 tablet_bootstrap.cc:654] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:46.999977 7161 tablet_bootstrap.cc:492] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:47.000285 7161 ts_tablet_manager.cc:1367] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.022s
I20240503 08:12:47.001567 7161 raft_consensus.cc:348] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:47.002063 7161 raft_consensus.cc:374] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.002255 7161 raft_consensus.cc:729] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.002671 7161 consensus_queue.cc:260] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:47.007586 7161 ts_tablet_manager.cc:1398] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:12:47.010154 7154 tablet_bootstrap.cc:492] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:47.010447 7154 ts_tablet_manager.cc:1367] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.007s
I20240503 08:12:47.012157 7154 raft_consensus.cc:348] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.012172 1671 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:47.012722 7154 raft_consensus.cc:374] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.012866 7154 raft_consensus.cc:729] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.013360 7154 consensus_queue.cc:260] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.016434 7154 ts_tablet_manager.cc:1398] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:47.020690 7163 tablet_bootstrap.cc:654] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.028100 4814 tablet_service.cc:1467] Processing CreateTablet for tablet af0c218defbe409583463058c4777414 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55b7a5a0a1904aeaa8656766b1711ea8]), partition=
I20240503 08:12:47.028429 4946 tablet_service.cc:1467] Processing CreateTablet for tablet af0c218defbe409583463058c4777414 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55b7a5a0a1904aeaa8656766b1711ea8]), partition=
I20240503 08:12:47.028553 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0c218defbe409583463058c4777414. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.028870 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0c218defbe409583463058c4777414. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.033979 4683 tablet_service.cc:1467] Processing CreateTablet for tablet af0c218defbe409583463058c4777414 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55b7a5a0a1904aeaa8656766b1711ea8]), partition=
I20240503 08:12:47.034408 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0c218defbe409583463058c4777414. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.036160 7056 raft_consensus.cc:2932] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:47.037086 7163 tablet_bootstrap.cc:492] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:47.037249 7156 tablet_bootstrap.cc:492] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:47.037257 7163 ts_tablet_manager.cc:1367] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.002s
I20240503 08:12:47.038232 7163 raft_consensus.cc:348] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:47.038488 7163 raft_consensus.cc:374] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.038599 7163 raft_consensus.cc:729] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.039034 7156 tablet_bootstrap.cc:654] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.038903 7163 consensus_queue.cc:260] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:47.040313 7159 tablet_bootstrap.cc:492] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:47.040539 7159 ts_tablet_manager.cc:1367] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.000s
I20240503 08:12:47.041630 7159 raft_consensus.cc:348] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.042022 7159 raft_consensus.cc:374] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.042155 7159 raft_consensus.cc:729] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.041994 1943 raft_consensus.cc:2932] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:47.042619 7159 consensus_queue.cc:260] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.044574 7154 tablet_bootstrap.cc:492] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:47.047483 7154 tablet_bootstrap.cc:654] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.054117 7159 ts_tablet_manager.cc:1398] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 08:12:47.056157 7156 tablet_bootstrap.cc:492] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:47.056504 7156 ts_tablet_manager.cc:1367] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.008s
I20240503 08:12:47.057595 7156 raft_consensus.cc:348] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.057919 7156 raft_consensus.cc:374] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.058063 7156 raft_consensus.cc:729] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.060740 7156 consensus_queue.cc:260] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.062959 7156 ts_tablet_manager.cc:1398] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:12:47.071285 7157 ts_tablet_manager.cc:1888] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:47.084347 7164 ts_tablet_manager.cc:1888] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:47.084620 7164 log.cc:1196] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/575dd6e43e3e41d294b6eb27ef7c68d0
I20240503 08:12:47.085242 7164 ts_tablet_manager.cc:1909] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:47.089900 7163 ts_tablet_manager.cc:1398] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.053s user 0.001s sys 0.001s
I20240503 08:12:47.091142 7163 tablet_bootstrap.cc:492] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:47.083943 1672 catalog_manager.cc:5484] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:47.091588 7155 tablet_bootstrap.cc:492] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:47.092702 7157 log.cc:1196] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/575dd6e43e3e41d294b6eb27ef7c68d0
I20240503 08:12:47.092777 7155 ts_tablet_manager.cc:1367] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.144s user 0.024s sys 0.043s
I20240503 08:12:47.094489 7157 ts_tablet_manager.cc:1909] T 575dd6e43e3e41d294b6eb27ef7c68d0 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:47.094949 7155 raft_consensus.cc:348] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:47.095494 7155 raft_consensus.cc:374] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.095808 7155 raft_consensus.cc:729] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.095865 7163 tablet_bootstrap.cc:654] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.095893 7154 tablet_bootstrap.cc:492] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:47.096055 7154 ts_tablet_manager.cc:1367] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.013s
I20240503 08:12:47.096465 7155 consensus_queue.cc:260] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:47.097025 7154 raft_consensus.cc:348] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.097306 7154 raft_consensus.cc:374] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.097445 7154 raft_consensus.cc:729] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.097759 7154 consensus_queue.cc:260] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.099757 7154 ts_tablet_manager.cc:1398] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:47.099519 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (table rebalancer_dynamic_table_new_name [id=90d6d1c2e2394f03b8238399f8666a63]) successfully deleted
I20240503 08:12:47.101311 7155 ts_tablet_manager.cc:1398] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:47.104286 7101 raft_consensus.cc:979] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: : Instructing follower 00b61625d8a7460892c118a2570afda5 to start an election
I20240503 08:12:47.104496 7101 raft_consensus.cc:1067] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Signalling peer 00b61625d8a7460892c118a2570afda5 to start an election
I20240503 08:12:47.105490 4966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f3123fdbb5c945bd84f97fb4c35c5523"
dest_uuid: "00b61625d8a7460892c118a2570afda5"
from {username='slave'} at 127.27.157.194:41645
I20240503 08:12:47.105737 4966 raft_consensus.cc:482] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:47.105875 4966 raft_consensus.cc:3037] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:47.107851 7163 tablet_bootstrap.cc:492] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:47.108120 7163 ts_tablet_manager.cc:1367] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20240503 08:12:47.108960 4966 raft_consensus.cc:504] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:47.108996 7163 raft_consensus.cc:348] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:47.109282 7163 raft_consensus.cc:374] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.109354 7163 raft_consensus.cc:729] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.109608 7163 consensus_queue.cc:260] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:47.110216 7163 ts_tablet_manager.cc:1398] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:47.110680 4966 leader_election.cc:288] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:47.112229 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3123fdbb5c945bd84f97fb4c35c5523" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:47.112568 1813 raft_consensus.cc:3037] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:47.115417 7116 raft_consensus.cc:482] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:47.115639 7116 raft_consensus.cc:504] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.116603 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3123fdbb5c945bd84f97fb4c35c5523" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:47.116765 7116 leader_election.cc:288] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:47.116914 1942 raft_consensus.cc:3032] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:47.117012 1942 raft_consensus.cc:729] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:47.117336 1942 consensus_queue.cc:260] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:47.117991 1942 raft_consensus.cc:3037] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:47.120565 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40476d1762154f6d9ce4855872492573" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:47.120812 2072 raft_consensus.cc:2454] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:47.121348 4636 leader_election.cc:302] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:47.121766 7116 raft_consensus.cc:2781] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:47.121881 7116 raft_consensus.cc:482] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:47.121954 7116 raft_consensus.cc:3037] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.122848 1813 raft_consensus.cc:2454] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 2.
I20240503 08:12:47.123934 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 575dd6e43e3e41d294b6eb27ef7c68d0 (table rebalancer_dynamic_table_new_name [id=90d6d1c2e2394f03b8238399f8666a63]) successfully deleted
I20240503 08:12:47.125056 7116 raft_consensus.cc:504] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.126096 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3123fdbb5c945bd84f97fb4c35c5523" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:47.126375 2202 raft_consensus.cc:3037] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:47.127377 1810 raft_consensus.cc:2932] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:47.130090 2202 raft_consensus.cc:2454] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 2.
I20240503 08:12:47.138418 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40476d1762154f6d9ce4855872492573" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:47.138684 4965 raft_consensus.cc:3037] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.141610 4965 raft_consensus.cc:2454] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:47.142580 4636 leader_election.cc:302] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:47.131111 7051 raft_consensus.cc:482] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:47.143546 7051 raft_consensus.cc:504] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:47.144342 7051 leader_election.cc:288] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:47.145200 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe20c189635648f5b78905e8b9643bcd" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:47.145499 4701 raft_consensus.cc:2454] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:47.146605 2140 leader_election.cc:302] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:47.147863 7115 raft_consensus.cc:2781] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:47.148386 7129 raft_consensus.cc:2781] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:47.148514 7129 raft_consensus.cc:482] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:47.148653 7129 raft_consensus.cc:3037] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.154127 7129 raft_consensus.cc:504] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:47.155952 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe20c189635648f5b78905e8b9643bcd" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:47.156316 4701 raft_consensus.cc:3037] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.160873 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40476d1762154f6d9ce4855872492573" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:47.161114 2072 raft_consensus.cc:3037] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.161865 7116 leader_election.cc:288] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:47.162964 4901 leader_election.cc:302] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:47.163388 7047 raft_consensus.cc:2781] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:47.164187 2072 raft_consensus.cc:2454] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:47.165725 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe20c189635648f5b78905e8b9643bcd" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:47.165947 1943 raft_consensus.cc:2454] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:47.175534 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40476d1762154f6d9ce4855872492573" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:47.176100 4966 raft_consensus.cc:2362] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1. Re-sending same reply.
I20240503 08:12:47.179888 4701 raft_consensus.cc:2454] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:47.180670 2140 leader_election.cc:302] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:47.181203 7051 raft_consensus.cc:2781] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:47.181367 7051 raft_consensus.cc:686] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:47.181727 7051 consensus_queue.cc:237] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:47.179224 1942 raft_consensus.cc:2454] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 2.
I20240503 08:12:47.195185 7129 leader_election.cc:288] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:47.196321 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe20c189635648f5b78905e8b9643bcd" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:47.196821 1942 raft_consensus.cc:3037] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.199811 7047 raft_consensus.cc:686] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:47.201570 1942 raft_consensus.cc:2454] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:47.206136 7115 raft_consensus.cc:686] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:47.206630 7115 consensus_queue.cc:237] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.207377 7173 tablet_bootstrap.cc:492] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:47.209683 7173 tablet_bootstrap.cc:654] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:47.200614 7047 consensus_queue.cc:237] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:47.213588 1672 catalog_manager.cc:5484] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:47.218945 1671 catalog_manager.cc:5484] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 1 to 2, leader changed from 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:47.230445 1671 catalog_manager.cc:5484] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:47.231703 7173 tablet_bootstrap.cc:492] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:47.233642 7173 ts_tablet_manager.cc:1367] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.005s
I20240503 08:12:47.235394 7173 raft_consensus.cc:348] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.235766 7173 raft_consensus.cc:374] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:47.235900 7173 raft_consensus.cc:729] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:47.236235 7173 consensus_queue.cc:260] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.238204 7173 ts_tablet_manager.cc:1398] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:12:47.248703 7062 raft_consensus.cc:482] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:47.248900 7062 raft_consensus.cc:504] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.249835 7062 leader_election.cc:288] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:47.251016 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0c218defbe409583463058c4777414" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:47.251298 4701 raft_consensus.cc:2454] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:47.252013 4770 leader_election.cc:302] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:47.252846 7061 raft_consensus.cc:2781] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:47.252990 7061 raft_consensus.cc:482] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:47.253084 7061 raft_consensus.cc:3037] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.256891 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0c218defbe409583463058c4777414" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:47.257191 4965 raft_consensus.cc:2454] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:47.260937 7061 raft_consensus.cc:504] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.267203 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0c218defbe409583463058c4777414" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:47.267500 4966 raft_consensus.cc:3037] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.267640 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0c218defbe409583463058c4777414" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:47.267846 4701 raft_consensus.cc:3037] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.271819 7061 leader_election.cc:288] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:47.272491 4701 raft_consensus.cc:2454] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:47.273010 4966 raft_consensus.cc:2454] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:47.273200 4770 leader_election.cc:302] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:47.274070 7061 raft_consensus.cc:2781] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:47.274837 7061 raft_consensus.cc:686] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:47.275410 7061 consensus_queue.cc:237] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:47.295418 1671 catalog_manager.cc:5484] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:47.323133 1810 consensus_queue.cc:237] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:47.332979 2202 raft_consensus.cc:1261] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:47.333045 1749 consensus_peers.cc:485] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc4d5ddaaa27463cba31eed19c4c1be0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:47.335091 6624 consensus_queue.cc:1035] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:47.343712 2072 raft_consensus.cc:1261] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:47.343995 7182 raft_consensus.cc:482] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:47.344194 7182 raft_consensus.cc:504] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:47.344731 7197 consensus_queue.cc:1035] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:47.345254 7182 leader_election.cc:288] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:47.346618 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6862b07fa36f4a458913803367c3f23e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:47.346881 4567 raft_consensus.cc:2454] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:47.347664 4900 leader_election.cc:302] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:47.348085 7182 raft_consensus.cc:2781] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:47.348232 7182 raft_consensus.cc:482] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:47.348323 7182 raft_consensus.cc:3037] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.352303 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6862b07fa36f4a458913803367c3f23e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:47.353843 7182 raft_consensus.cc:504] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:47.354305 2202 raft_consensus.cc:2454] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:47.357627 7197 raft_consensus.cc:2932] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:47.362075 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6862b07fa36f4a458913803367c3f23e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:47.362499 2202 raft_consensus.cc:3037] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.364074 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6862b07fa36f4a458913803367c3f23e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:47.364342 4567 raft_consensus.cc:3037] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:47.365607 7182 leader_election.cc:288] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:47.367247 2202 raft_consensus.cc:2454] T 6862b07fa36f4a458913803367c3f23e P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:47.368070 4901 leader_election.cc:302] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 184c543cb05d49cc98ba1c56617be7a5; no voters:
I20240503 08:12:47.366886 2072 raft_consensus.cc:2932] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:47.368901 7182 raft_consensus.cc:2781] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:47.369077 7182 raft_consensus.cc:686] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:47.368707 2204 raft_consensus.cc:2932] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:47.369400 7182 consensus_queue.cc:237] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:47.375700 4567 raft_consensus.cc:2454] T 6862b07fa36f4a458913803367c3f23e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:47.376361 1665 catalog_manager.cc:5484] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:47.389856 1672 catalog_manager.cc:5484] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:47.406133 4965 consensus_queue.cc:237] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:47.410702 4701 raft_consensus.cc:1261] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:47.410825 2204 raft_consensus.cc:1261] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:12:47.411427 4900 consensus_peers.cc:485] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8737ef0936c641ccaf0b804b1d9c3e07. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:47.412017 7184 consensus_queue.cc:1035] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:47.413324 7186 consensus_queue.cc:1035] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:47.416318 7184 raft_consensus.cc:2932] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:47.417155 4701 raft_consensus.cc:2932] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:47.418468 7211 ts_tablet_manager.cc:902] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:47.424121 1665 catalog_manager.cc:5484] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:47.427685 2204 raft_consensus.cc:2932] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:47.439682 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:47.445506 7211 tablet_copy_client.cc:308] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.196:42183
I20240503 08:12:47.446624 2224 tablet_copy_service.cc:140] P 184c543cb05d49cc98ba1c56617be7a5: Received BeginTabletCopySession request for tablet dc342fed6c6343a080ae0ff2c53cb140 from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:51901)
I20240503 08:12:47.446884 2224 tablet_copy_service.cc:161] P 184c543cb05d49cc98ba1c56617be7a5: Beginning new tablet copy session on tablet dc342fed6c6343a080ae0ff2c53cb140 from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:51901: session id = 31f41542452b4b1eb994ff4223d039fc-dc342fed6c6343a080ae0ff2c53cb140
I20240503 08:12:47.452034 2224 tablet_copy_source_session.cc:205] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:47.453961 7211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc342fed6c6343a080ae0ff2c53cb140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.461948 7211 tablet_copy_client.cc:783] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:47.462529 7211 tablet_copy_client.cc:647] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:47.464799 7211 tablet_copy_client.cc:523] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:47.467226 4701 raft_consensus.cc:1261] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:47.467854 4965 raft_consensus.cc:1261] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:47.468569 6947 consensus_queue.cc:1035] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.468899 7211 tablet_bootstrap.cc:492] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:47.468923 7062 consensus_queue.cc:1035] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.487073 1672 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet af0c218defbe409583463058c4777414 (table rebalancer_dynamic_table [id=55b7a5a0a1904aeaa8656766b1711ea8]). Expected version 1 got 0
I20240503 08:12:47.492084 1672 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet af0c218defbe409583463058c4777414 (table rebalancer_dynamic_table [id=55b7a5a0a1904aeaa8656766b1711ea8]). Expected version 1 got 0
I20240503 08:12:47.492949 7228 tablet.cc:1722] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:47.494720 7228 tablet.cc:1620] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:47.494674 7227 tablet.cc:1722] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:47.495517 1672 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet af0c218defbe409583463058c4777414 (table rebalancer_dynamic_table [id=55b7a5a0a1904aeaa8656766b1711ea8]). Expected version 1 got 0
I20240503 08:12:47.495831 7227 tablet.cc:1620] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:47.499933 1665 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet af0c218defbe409583463058c4777414 (table rebalancer_dynamic_table [id=55b7a5a0a1904aeaa8656766b1711ea8]). Expected version 1 got 0
I20240503 08:12:47.502174 1672 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet af0c218defbe409583463058c4777414 (table rebalancer_dynamic_table [id=55b7a5a0a1904aeaa8656766b1711ea8]). Expected version 1 got 0
I20240503 08:12:47.487221 7226 tablet.cc:1722] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:47.503121 7226 tablet.cc:1620] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:47.505103 1671 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=55b7a5a0a1904aeaa8656766b1711ea8] alter complete (version 1)
I20240503 08:12:47.541127 2072 consensus_queue.cc:237] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:47.544183 2204 raft_consensus.cc:1261] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:47.544217 1942 raft_consensus.cc:1261] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:47.544909 7109 consensus_queue.cc:1035] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:47.545300 7055 consensus_queue.cc:1035] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:12:47.546115 2010 consensus_peers.cc:485] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b00417d7369a46aabfa933109c645c2c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:47.550751 7109 raft_consensus.cc:2932] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:47.555267 2202 raft_consensus.cc:2932] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:47.558915 1671 catalog_manager.cc:5484] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:47.565742 7178 consensus_queue.cc:1035] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.575470 7211 tablet_bootstrap.cc:492] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:47.576056 7211 tablet_bootstrap.cc:492] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:12:47.576395 7211 ts_tablet_manager.cc:1367] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.108s user 0.058s sys 0.001s
I20240503 08:12:47.577872 7211 raft_consensus.cc:348] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:47.578182 7211 raft_consensus.cc:729] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:12:47.578428 7211 consensus_queue.cc:260] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:47.590592 7211 ts_tablet_manager.cc:1398] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240503 08:12:47.591872 2224 tablet_copy_service.cc:342] P 184c543cb05d49cc98ba1c56617be7a5: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-dc342fed6c6343a080ae0ff2c53cb140 received from {username='slave'} at 127.27.157.198:51901
I20240503 08:12:47.592056 2224 tablet_copy_service.cc:433] P 184c543cb05d49cc98ba1c56617be7a5: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-dc342fed6c6343a080ae0ff2c53cb140 on tablet dc342fed6c6343a080ae0ff2c53cb140 with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:47.596451 1941 raft_consensus.cc:2932] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:47.614122 1672 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:47.623590 7226 tablet.cc:1722] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:47.624191 7226 tablet.cc:1620] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:47.624523 7227 tablet.cc:1722] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:47.625106 7227 tablet.cc:1620] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:47.631588 7228 tablet.cc:1722] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:47.632165 7228 tablet.cc:1620] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:47.640789 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=55b7a5a0a1904aeaa8656766b1711ea8] alter complete (version 2)
I20240503 08:12:47.643721 7051 consensus_queue.cc:1035] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.705682 7115 consensus_queue.cc:1035] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.717278 7180 consensus_queue.cc:1035] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.757762 1671 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:47.768678 7226 tablet.cc:1722] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:47.771651 7228 tablet.cc:1722] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:47.772454 1810 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:12:47.773139 7184 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:47.773723 7227 tablet.cc:1722] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:47.784344 1943 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:12:47.785140 7184 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:47.794382 1671 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=55b7a5a0a1904aeaa8656766b1711ea8] alter complete (version 3)
I20240503 08:12:47.797341 7184 consensus_queue.cc:1035] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.804523 7186 consensus_queue.cc:1035] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:47.834748 2202 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 08:12:47.835492 7186 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:47.868791 4965 consensus_queue.cc:237] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:47.870713 2202 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:47.870846 1810 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 08:12:47.871438 7184 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:47.872014 7184 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:47.874305 7184 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:47.874946 1810 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:47.874962 2202 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:47.879122 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f3123fdbb5c945bd84f97fb4c35c5523 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:47.880556 1671 catalog_manager.cc:5484] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 5 to 7, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:47.884881 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet f3123fdbb5c945bd84f97fb4c35c5523 with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:47.889854 7281 tablet_replica.cc:330] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:47.892484 7281 raft_consensus.cc:2229] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:47.892755 7281 raft_consensus.cc:2258] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:47.893625 7281 ts_tablet_manager.cc:1875] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:47.897943 1671 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:47.898190 1671 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:47.903837 1671 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet af0c218defbe409583463058c4777414
I20240503 08:12:47.905084 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet af0c218defbe409583463058c4777414 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:47 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:47.905521 4946 tablet_service.cc:1514] Processing DeleteTablet for tablet af0c218defbe409583463058c4777414 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:47 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:47.905735 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet af0c218defbe409583463058c4777414 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:47 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:47.906585 7286 tablet_replica.cc:330] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:47.906877 7286 raft_consensus.cc:2229] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:47.907202 7286 raft_consensus.cc:2258] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:47.908655 7281 ts_tablet_manager.cc:1888] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 08:12:47.908849 7281 log.cc:1196] T f3123fdbb5c945bd84f97fb4c35c5523 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/f3123fdbb5c945bd84f97fb4c35c5523
I20240503 08:12:47.909353 7284 tablet_replica.cc:330] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:47.909587 7284 raft_consensus.cc:2229] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:47.909927 7284 raft_consensus.cc:2258] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:47.910203 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet f3123fdbb5c945bd84f97fb4c35c5523 (table rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]) successfully deleted
I20240503 08:12:47.910367 7285 tablet_replica.cc:330] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:47.910602 7285 raft_consensus.cc:2229] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:47.910856 7285 raft_consensus.cc:2258] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:47.908636 7286 ts_tablet_manager.cc:1875] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:47.911402 7284 ts_tablet_manager.cc:1875] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:47.912932 7285 ts_tablet_manager.cc:1875] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:47.915650 4701 raft_consensus.cc:1203] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:47.924252 7291 ts_tablet_manager.cc:902] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:47.931748 7284 ts_tablet_manager.cc:1888] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:47.932099 7284 log.cc:1196] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/af0c218defbe409583463058c4777414
I20240503 08:12:47.932672 7284 ts_tablet_manager.cc:1909] T af0c218defbe409583463058c4777414 P f24648c92c384f1ba349eee801f985e2: Deleting consensus metadata
I20240503 08:12:47.933504 7286 ts_tablet_manager.cc:1888] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:47.934217 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet af0c218defbe409583463058c4777414 (table rebalancer_dynamic_table_new_name [id=55b7a5a0a1904aeaa8656766b1711ea8]) successfully deleted
I20240503 08:12:47.942430 7290 ts_tablet_manager.cc:902] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:47.943567 7286 log.cc:1196] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/af0c218defbe409583463058c4777414
I20240503 08:12:47.947110 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:47.952216 7286 ts_tablet_manager.cc:1909] T af0c218defbe409583463058c4777414 P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:47.952593 7285 ts_tablet_manager.cc:1888] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:47.952844 7285 log.cc:1196] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/af0c218defbe409583463058c4777414
I20240503 08:12:47.953380 7285 ts_tablet_manager.cc:1909] T af0c218defbe409583463058c4777414 P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:47.954684 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet af0c218defbe409583463058c4777414 (table rebalancer_dynamic_table_new_name [id=55b7a5a0a1904aeaa8656766b1711ea8]) successfully deleted
I20240503 08:12:47.961160 1791 tablet_service.cc:1467] Processing CreateTablet for tablet c4e697bc477a4899b69e4fdcc4e235e7 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9bfbde099d3143e2b4a26a7c01253ea3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.961664 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e697bc477a4899b69e4fdcc4e235e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.961678 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet af0c218defbe409583463058c4777414 (table rebalancer_dynamic_table_new_name [id=55b7a5a0a1904aeaa8656766b1711ea8]) successfully deleted
I20240503 08:12:47.961817 4683 tablet_service.cc:1467] Processing CreateTablet for tablet c4e697bc477a4899b69e4fdcc4e235e7 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9bfbde099d3143e2b4a26a7c01253ea3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.962328 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e697bc477a4899b69e4fdcc4e235e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.968556 4548 tablet_service.cc:1467] Processing CreateTablet for tablet c4e697bc477a4899b69e4fdcc4e235e7 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9bfbde099d3143e2b4a26a7c01253ea3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.969110 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4e697bc477a4899b69e4fdcc4e235e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.982954 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 4e09aca7245948c697579899ad5bfd96 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9bfbde099d3143e2b4a26a7c01253ea3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.983577 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e09aca7245948c697579899ad5bfd96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.990460 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 4e09aca7245948c697579899ad5bfd96 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9bfbde099d3143e2b4a26a7c01253ea3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.990692 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 4e09aca7245948c697579899ad5bfd96 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9bfbde099d3143e2b4a26a7c01253ea3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.991014 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e09aca7245948c697579899ad5bfd96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.991096 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e09aca7245948c697579899ad5bfd96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:47.997627 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 6d69a5126dff43af8ba3acc931d17799 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9bfbde099d3143e2b4a26a7c01253ea3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:47.998095 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d69a5126dff43af8ba3acc931d17799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.001137 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 6d69a5126dff43af8ba3acc931d17799 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9bfbde099d3143e2b4a26a7c01253ea3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.001624 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d69a5126dff43af8ba3acc931d17799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.007877 1671 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:48.022015 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 6d69a5126dff43af8ba3acc931d17799 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9bfbde099d3143e2b4a26a7c01253ea3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:48.022627 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d69a5126dff43af8ba3acc931d17799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.028345 7305 tablet_bootstrap.cc:492] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:48.034747 4549 tablet_service.cc:1467] Processing CreateTablet for tablet 3f91e3e73cf34ecbb80faab9025cb115 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32579d5f9fd04dd08d2d5f17cd9ef5bf]), partition=
I20240503 08:12:48.035871 4681 tablet_service.cc:1467] Processing CreateTablet for tablet 3f91e3e73cf34ecbb80faab9025cb115 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32579d5f9fd04dd08d2d5f17cd9ef5bf]), partition=
I20240503 08:12:48.036468 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f91e3e73cf34ecbb80faab9025cb115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.039958 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 3f91e3e73cf34ecbb80faab9025cb115 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32579d5f9fd04dd08d2d5f17cd9ef5bf]), partition=
I20240503 08:12:48.040484 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f91e3e73cf34ecbb80faab9025cb115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.040527 7309 tablet_bootstrap.cc:492] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:48.044169 7304 ts_tablet_manager.cc:902] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:48.044665 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f91e3e73cf34ecbb80faab9025cb115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.045217 7305 tablet_bootstrap.cc:654] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.059857 7306 tablet_bootstrap.cc:492] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:48.062873 7306 tablet_bootstrap.cc:654] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.064396 7308 tablet_bootstrap.cc:492] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:48.066955 7308 tablet_bootstrap.cc:654] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.067801 7307 tablet_bootstrap.cc:492] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:48.069468 7309 tablet_bootstrap.cc:654] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.069978 7307 tablet_bootstrap.cc:654] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.071367 7310 tablet_bootstrap.cc:492] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:48.074822 7310 tablet_bootstrap.cc:654] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.081372 7305 tablet_bootstrap.cc:492] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:48.079885 7304 tablet_copy_client.cc:308] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:12:48.086568 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet 8737ef0936c641ccaf0b804b1d9c3e07 from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:45051)
I20240503 08:12:48.086779 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet 8737ef0936c641ccaf0b804b1d9c3e07 from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:45051: session id = 5f95b05d111e4090bbc59dec394ed0ab-8737ef0936c641ccaf0b804b1d9c3e07
I20240503 08:12:48.087956 7306 tablet_bootstrap.cc:492] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:48.089854 4988 tablet_copy_source_session.cc:205] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:48.093015 7304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8737ef0936c641ccaf0b804b1d9c3e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.093628 7305 ts_tablet_manager.cc:1367] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.000s
I20240503 08:12:48.096326 7302 tablet_bootstrap.cc:492] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:48.097383 7306 ts_tablet_manager.cc:1367] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.002s
I20240503 08:12:48.098345 7306 raft_consensus.cc:348] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:48.098675 7306 raft_consensus.cc:374] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.098765 7306 raft_consensus.cc:729] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.099062 7306 consensus_queue.cc:260] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:48.099360 7309 tablet_bootstrap.cc:492] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:48.099722 7305 raft_consensus.cc:348] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:48.100584 7305 raft_consensus.cc:374] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.100755 7305 raft_consensus.cc:729] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.100452 7290 tablet_copy_client.cc:308] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:12:48.101066 7302 tablet_bootstrap.cc:654] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.101464 7309 ts_tablet_manager.cc:1367] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.006s
I20240503 08:12:48.101862 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet b00417d7369a46aabfa933109c645c2c from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:52311)
I20240503 08:12:48.101816 7305 consensus_queue.cc:260] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:48.102044 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet b00417d7369a46aabfa933109c645c2c from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:52311: session id = f24648c92c384f1ba349eee801f985e2-b00417d7369a46aabfa933109c645c2c
I20240503 08:12:48.103156 7199 raft_consensus.cc:482] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.103358 7199 raft_consensus.cc:504] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:48.103713 7305 ts_tablet_manager.cc:1398] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:12:48.104003 7305 tablet_bootstrap.cc:492] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:48.104199 7199 leader_election.cc:288] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:48.104938 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d69a5126dff43af8ba3acc931d17799" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:48.105749 2093 tablet_copy_source_session.cc:205] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 08:12:48.105995 1747 leader_election.cc:341] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:48.106539 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d69a5126dff43af8ba3acc931d17799" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
W20240503 08:12:48.107313 1748 leader_election.cc:341] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:48.107455 1748 leader_election.cc:302] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f581390da449ccb9e299011296405e; no voters: 00b61625d8a7460892c118a2570afda5, 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:12:48.108197 7199 raft_consensus.cc:2726] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:48.103282 7309 raft_consensus.cc:348] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:48.109840 7309 raft_consensus.cc:374] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.111363 7290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00417d7369a46aabfa933109c645c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.114117 7305 tablet_bootstrap.cc:654] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.115757 7306 ts_tablet_manager.cc:1398] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240503 08:12:48.119206 7306 tablet_bootstrap.cc:492] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:48.121414 7302 tablet_bootstrap.cc:492] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:48.121619 7302 ts_tablet_manager.cc:1367] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.007s
I20240503 08:12:48.122354 7309 raft_consensus.cc:729] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.122548 7302 raft_consensus.cc:348] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.122885 7302 raft_consensus.cc:374] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.122793 7309 consensus_queue.cc:260] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:48.123014 7302 raft_consensus.cc:729] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.123389 7302 consensus_queue.cc:260] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.124081 7302 ts_tablet_manager.cc:1398] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:12:48.124441 7302 tablet_bootstrap.cc:492] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:48.125550 7290 tablet_copy_client.cc:783] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:48.126325 7290 tablet_copy_client.cc:647] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:48.126426 7302 tablet_bootstrap.cc:654] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.131718 7290 tablet_copy_client.cc:523] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:48.128748 7309 ts_tablet_manager.cc:1398] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20240503 08:12:48.147018 7309 tablet_bootstrap.cc:492] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:48.148739 7291 tablet_copy_client.cc:308] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:41181
I20240503 08:12:48.150609 7305 tablet_bootstrap.cc:492] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:48.150911 7305 ts_tablet_manager.cc:1367] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.002s
I20240503 08:12:48.151197 1833 tablet_copy_service.cc:140] P b8f581390da449ccb9e299011296405e: Received BeginTabletCopySession request for tablet fc4d5ddaaa27463cba31eed19c4c1be0 from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:57293)
I20240503 08:12:48.151374 1833 tablet_copy_service.cc:161] P b8f581390da449ccb9e299011296405e: Beginning new tablet copy session on tablet fc4d5ddaaa27463cba31eed19c4c1be0 from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:57293: session id = f24648c92c384f1ba349eee801f985e2-fc4d5ddaaa27463cba31eed19c4c1be0
I20240503 08:12:48.152082 7305 raft_consensus.cc:348] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:48.152307 7305 raft_consensus.cc:374] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.152381 7305 raft_consensus.cc:729] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.154103 7305 consensus_queue.cc:260] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:48.155901 1833 tablet_copy_source_session.cc:205] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:48.156191 7305 ts_tablet_manager.cc:1398] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:48.159411 7291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4d5ddaaa27463cba31eed19c4c1be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.166150 7308 tablet_bootstrap.cc:492] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:48.166357 7308 ts_tablet_manager.cc:1367] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.102s user 0.005s sys 0.000s
I20240503 08:12:48.166524 7309 tablet_bootstrap.cc:654] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.168604 7310 tablet_bootstrap.cc:492] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:48.168884 7310 ts_tablet_manager.cc:1367] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.098s user 0.006s sys 0.000s
I20240503 08:12:48.172204 7308 raft_consensus.cc:348] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:48.173717 7310 raft_consensus.cc:348] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.174139 7308 raft_consensus.cc:374] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.174018 7310 raft_consensus.cc:374] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.174357 7310 raft_consensus.cc:729] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.174916 7306 tablet_bootstrap.cc:654] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.175057 7308 raft_consensus.cc:729] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.176970 7310 consensus_queue.cc:260] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.177462 7308 consensus_queue.cc:260] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:48.179605 7307 tablet_bootstrap.cc:492] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:48.179817 7307 ts_tablet_manager.cc:1367] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.112s user 0.006s sys 0.000s
I20240503 08:12:48.180668 7307 raft_consensus.cc:348] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:48.180969 7307 raft_consensus.cc:374] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.181128 7307 raft_consensus.cc:729] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.181434 7307 consensus_queue.cc:260] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:48.186206 7302 tablet_bootstrap.cc:492] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:48.186524 7302 ts_tablet_manager.cc:1367] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.062s user 0.017s sys 0.013s
I20240503 08:12:48.187685 7302 raft_consensus.cc:348] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.188077 7302 raft_consensus.cc:374] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.188207 7302 raft_consensus.cc:729] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.188566 7302 consensus_queue.cc:260] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.188776 7307 ts_tablet_manager.cc:1398] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:12:48.189461 7310 ts_tablet_manager.cc:1398] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 08:12:48.190088 7308 ts_tablet_manager.cc:1398] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20240503 08:12:48.190526 6375 raft_consensus.cc:482] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.190690 6375 raft_consensus.cc:504] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:48.191951 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d69a5126dff43af8ba3acc931d17799" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:48.192186 1810 raft_consensus.cc:2454] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:48.192684 1879 leader_election.cc:302] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:48.193750 7241 raft_consensus.cc:2781] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.193888 7241 raft_consensus.cc:482] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.194005 7241 raft_consensus.cc:3037] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.195353 7310 tablet_bootstrap.cc:492] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:48.197600 7241 raft_consensus.cc:504] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:48.198987 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d69a5126dff43af8ba3acc931d17799" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:48.199304 4965 raft_consensus.cc:3037] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.200435 6375 leader_election.cc:288] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:48.201297 7241 leader_election.cc:288] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:48.201375 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d69a5126dff43af8ba3acc931d17799" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:48.200799 7302 ts_tablet_manager.cc:1398] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 08:12:48.202486 4965 raft_consensus.cc:2454] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:48.202966 1878 leader_election.cc:302] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:48.177294 7290 tablet_bootstrap.cc:492] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:48.205539 7241 raft_consensus.cc:2781] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.205788 7241 raft_consensus.cc:686] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:48.206130 7241 consensus_queue.cc:237] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:48.206928 7304 tablet_copy_client.cc:783] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:48.207305 7304 tablet_copy_client.cc:647] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:48.209264 7304 tablet_copy_client.cc:523] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:48.209403 7310 tablet_bootstrap.cc:654] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:48.211827 7306 tablet_bootstrap.cc:492] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:48.212005 7306 ts_tablet_manager.cc:1367] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.000s
I20240503 08:12:48.212743 7306 raft_consensus.cc:348] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.212957 7306 raft_consensus.cc:374] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.213011 7306 raft_consensus.cc:729] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.221802 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d69a5126dff43af8ba3acc931d17799" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:48.222081 1810 raft_consensus.cc:3037] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.224728 7306 consensus_queue.cc:260] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.225181 7309 tablet_bootstrap.cc:492] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:48.225376 7306 ts_tablet_manager.cc:1398] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 08:12:48.225800 1810 raft_consensus.cc:2454] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:48.226852 7309 ts_tablet_manager.cc:1367] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.080s user 0.000s sys 0.008s
I20240503 08:12:48.229064 7291 tablet_copy_client.cc:783] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:48.229494 7291 tablet_copy_client.cc:647] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:48.233983 1668 catalog_manager.cc:5484] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:48.237620 7291 tablet_copy_client.cc:523] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:48.232363 7241 raft_consensus.cc:482] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.238015 7241 raft_consensus.cc:504] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:48.239019 7241 leader_election.cc:288] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:48.239363 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e09aca7245948c697579899ad5bfd96" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:48.239364 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e09aca7245948c697579899ad5bfd96" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:48.239617 2072 raft_consensus.cc:2454] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:48.239661 4834 raft_consensus.cc:2454] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:48.240988 1878 leader_election.cc:302] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:48.241586 6375 raft_consensus.cc:2781] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.241673 6375 raft_consensus.cc:482] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.241735 6375 raft_consensus.cc:3037] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.241986 7291 tablet_bootstrap.cc:492] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:48.245321 7310 tablet_bootstrap.cc:492] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:48.245719 7310 ts_tablet_manager.cc:1367] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.004s
I20240503 08:12:48.244516 6375 raft_consensus.cc:504] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:48.237627 7309 raft_consensus.cc:348] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.246568 7309 raft_consensus.cc:374] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.246827 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e09aca7245948c697579899ad5bfd96" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:48.246827 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e09aca7245948c697579899ad5bfd96" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:48.247066 4834 raft_consensus.cc:3037] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.247066 2072 raft_consensus.cc:3037] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.249967 2072 raft_consensus.cc:2454] T 4e09aca7245948c697579899ad5bfd96 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:48.250530 1878 leader_election.cc:302] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:48.251686 4834 raft_consensus.cc:2454] T 4e09aca7245948c697579899ad5bfd96 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:48.252095 6375 leader_election.cc:288] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:48.252331 6375 raft_consensus.cc:2781] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.252569 6375 raft_consensus.cc:686] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:48.252882 6375 consensus_queue.cc:237] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:48.254721 7304 tablet_bootstrap.cc:492] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:48.255846 7310 raft_consensus.cc:348] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.256618 7310 raft_consensus.cc:374] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:48.256790 7310 raft_consensus.cc:729] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.257232 7310 consensus_queue.cc:260] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.264720 7310 ts_tablet_manager.cc:1398] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240503 08:12:48.246846 7309 raft_consensus.cc:729] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:48.275851 7309 consensus_queue.cc:260] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.276611 7309 ts_tablet_manager.cc:1398] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.049s user 0.002s sys 0.000s
I20240503 08:12:48.282394 1671 catalog_manager.cc:5484] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:48.307227 7304 tablet_bootstrap.cc:492] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:48.308537 7304 tablet_bootstrap.cc:492] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:48.309834 7304 ts_tablet_manager.cc:1367] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.005s
I20240503 08:12:48.311806 7066 raft_consensus.cc:482] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.311995 7066 raft_consensus.cc:504] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.312903 7066 leader_election.cc:288] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:48.313843 7304 raft_consensus.cc:348] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:48.314160 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f91e3e73cf34ecbb80faab9025cb115" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:48.314244 7304 raft_consensus.cc:729] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:48.314394 4834 raft_consensus.cc:2454] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:48.315093 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f91e3e73cf34ecbb80faab9025cb115" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:48.315358 4701 raft_consensus.cc:2454] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:48.314515 7304 consensus_queue.cc:260] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:48.316802 4504 leader_election.cc:302] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:48.319249 7066 raft_consensus.cc:2781] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.319382 7066 raft_consensus.cc:482] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.319499 7066 raft_consensus.cc:3037] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.323096 7066 raft_consensus.cc:504] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.324625 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f91e3e73cf34ecbb80faab9025cb115" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:48.324898 4834 raft_consensus.cc:3037] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.325124 7066 leader_election.cc:288] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:48.327180 7304 ts_tablet_manager.cc:1398] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 08:12:48.328164 4834 raft_consensus.cc:2454] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:48.328740 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f91e3e73cf34ecbb80faab9025cb115" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:48.329021 4701 raft_consensus.cc:3037] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.331924 4701 raft_consensus.cc:2454] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:48.336108 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-8737ef0936c641ccaf0b804b1d9c3e07 received from {username='slave'} at 127.27.157.197:45051
I20240503 08:12:48.336139 4504 leader_election.cc:302] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:48.336270 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-8737ef0936c641ccaf0b804b1d9c3e07 on tablet 8737ef0936c641ccaf0b804b1d9c3e07 with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:48.337383 7066 raft_consensus.cc:2781] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.340613 7066 raft_consensus.cc:686] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:48.341032 7066 consensus_queue.cc:237] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.359544 1671 catalog_manager.cc:5484] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:48.366247 1944 consensus_queue.cc:237] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
W20240503 08:12:48.377399 1878 consensus_peers.cc:485] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d11cd50b562408f9d3bc24a974bdba8. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:48.380889 7290 tablet_bootstrap.cc:492] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:48.381582 7290 tablet_bootstrap.cc:492] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:12:48.383735 7290 ts_tablet_manager.cc:1367] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.207s user 0.048s sys 0.012s
I20240503 08:12:48.384459 2072 raft_consensus.cc:1261] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:48.385054 2202 raft_consensus.cc:1261] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:48.385862 6375 consensus_queue.cc:1035] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:48.386173 7336 consensus_queue.cc:1035] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:48.387024 7291 tablet_bootstrap.cc:492] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:48.387622 7291 tablet_bootstrap.cc:492] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:12:48.387908 7291 ts_tablet_manager.cc:1367] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.146s user 0.048s sys 0.008s
I20240503 08:12:48.388916 7291 raft_consensus.cc:348] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:48.389320 7291 raft_consensus.cc:729] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:12:48.389631 7291 consensus_queue.cc:260] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:48.390435 7291 ts_tablet_manager.cc:1398] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:48.391834 1833 tablet_copy_service.cc:342] P b8f581390da449ccb9e299011296405e: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-fc4d5ddaaa27463cba31eed19c4c1be0 received from {username='slave'} at 127.27.157.199:57293
I20240503 08:12:48.391952 1833 tablet_copy_service.cc:433] P b8f581390da449ccb9e299011296405e: ending tablet copy session f24648c92c384f1ba349eee801f985e2-fc4d5ddaaa27463cba31eed19c4c1be0 on tablet fc4d5ddaaa27463cba31eed19c4c1be0 with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:12:48.393041 7336 raft_consensus.cc:2932] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:48.393453 7290 raft_consensus.cc:348] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:48.393772 7290 raft_consensus.cc:729] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:12:48.394034 7290 consensus_queue.cc:260] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:48.394678 7290 ts_tablet_manager.cc:1398] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.011s user 0.000s sys 0.000s
I20240503 08:12:48.396405 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-b00417d7369a46aabfa933109c645c2c received from {username='slave'} at 127.27.157.199:52311
I20240503 08:12:48.396562 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session f24648c92c384f1ba349eee801f985e2-b00417d7369a46aabfa933109c645c2c on tablet b00417d7369a46aabfa933109c645c2c with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:12:48.396628 2072 raft_consensus.cc:2932] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:48.406041 1668 catalog_manager.cc:5484] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:48.411911 2202 raft_consensus.cc:2932] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:48.430243 1944 consensus_queue.cc:237] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:48.432754 2072 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:48.432813 2202 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:12:48.433036 1877 consensus_peers.cc:485] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 674e558bf5db4f169f8cdcb08c40f5fb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:48.433327 6375 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:48.433658 7336 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:48.435268 4834 raft_consensus.cc:1203] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:48.445174 7336 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:48.448230 2073 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:48.449142 2202 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:48.454823 1671 catalog_manager.cc:5484] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 4, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:48.470800 4834 consensus_queue.cc:237] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
W20240503 08:12:48.474419 4768 consensus_peers.cc:485] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d4f6c5975f8446a9f6ba8cae137bda0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:48.475266 4701 raft_consensus.cc:1261] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:48.475354 1944 raft_consensus.cc:1261] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:48.476063 7062 consensus_queue.cc:1035] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:48.476429 6947 consensus_queue.cc:1035] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:48.480654 6947 raft_consensus.cc:2932] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:48.482759 4701 raft_consensus.cc:2932] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:48.482715 1944 raft_consensus.cc:2932] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:48.485927 7129 raft_consensus.cc:1050] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:12:48.486913 7129 consensus_queue.cc:237] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:48.492344 1668 catalog_manager.cc:5484] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:48.495427 7066 raft_consensus.cc:482] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:48.495626 7066 raft_consensus.cc:504] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.496551 7066 leader_election.cc:288] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:48.497092 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e697bc477a4899b69e4fdcc4e235e7" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:48.497318 1810 raft_consensus.cc:2454] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:48.497471 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e697bc477a4899b69e4fdcc4e235e7" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:48.497736 4701 raft_consensus.cc:2454] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:48.497809 4503 leader_election.cc:302] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:48.498158 7066 raft_consensus.cc:2781] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:48.498278 7066 raft_consensus.cc:482] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:48.498368 7066 raft_consensus.cc:3037] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.501255 4701 raft_consensus.cc:1261] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:48.501984 7051 consensus_queue.cc:1035] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:48.503994 4834 raft_consensus.cc:1203] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:48.506549 7066 raft_consensus.cc:504] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.508363 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e697bc477a4899b69e4fdcc4e235e7" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:48.508421 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e697bc477a4899b69e4fdcc4e235e7" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:48.508587 1810 raft_consensus.cc:3037] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.508648 4703 raft_consensus.cc:3037] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:48.511740 4703 raft_consensus.cc:2454] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:48.512620 1813 raft_consensus.cc:1261] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:48.512620 1944 raft_consensus.cc:1261] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:48.513216 7129 consensus_queue.cc:1035] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:48.513330 4504 leader_election.cc:302] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:48.513653 7354 raft_consensus.cc:2781] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:48.513789 7354 raft_consensus.cc:686] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:48.514124 7354 consensus_queue.cc:237] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:48.516094 7129 consensus_queue.cc:1035] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:48.517889 1810 raft_consensus.cc:2454] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:48.521283 1671 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:48.521472 7066 leader_election.cc:288] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:48.534911 7051 raft_consensus.cc:2932] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:48.535440 1671 catalog_manager.cc:5484] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:48.535600 1813 raft_consensus.cc:2932] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:48.543264 1668 catalog_manager.cc:5484] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:48.544103 1944 raft_consensus.cc:2932] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:48.555442 4703 raft_consensus.cc:2932] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:48.561949 4834 raft_consensus.cc:1261] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:48.563124 7066 consensus_queue.cc:1035] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 08:12:48.563874 4703 raft_consensus.cc:1261] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:48.565042 7066 consensus_queue.cc:1035] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.576228 2202 consensus_queue.cc:237] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:48.579412 4701 raft_consensus.cc:1261] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:48.580015 1813 raft_consensus.cc:1261] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:48.580245 7051 consensus_queue.cc:1035] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:48.580943 7129 consensus_queue.cc:1035] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:48.584637 7051 raft_consensus.cc:2932] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:48.587980 4701 raft_consensus.cc:2932] T dc342fed6c6343a080ae0ff2c53cb140 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:48.590683 1810 raft_consensus.cc:2932] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:48.592336 4567 raft_consensus.cc:1203] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:48.600275 1672 catalog_manager.cc:5484] T dc342fed6c6343a080ae0ff2c53cb140 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:48.603672 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc342fed6c6343a080ae0ff2c53cb140 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:48.604705 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet dc342fed6c6343a080ae0ff2c53cb140 with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:48.608407 1672 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet 3f91e3e73cf34ecbb80faab9025cb115 (table rebalancer_dynamic_table [id=32579d5f9fd04dd08d2d5f17cd9ef5bf]). Expected version 1 got 0
I20240503 08:12:48.618656 7396 tablet_replica.cc:330] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:48.619110 7396 raft_consensus.cc:2229] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:48.619607 7396 raft_consensus.cc:2258] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:48.620975 7396 ts_tablet_manager.cc:1875] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:48.634413 7396 ts_tablet_manager.cc:1888] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:48.634686 7396 log.cc:1196] T dc342fed6c6343a080ae0ff2c53cb140 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/dc342fed6c6343a080ae0ff2c53cb140
I20240503 08:12:48.635835 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet dc342fed6c6343a080ae0ff2c53cb140 (table rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]) successfully deleted
I20240503 08:12:48.636816 7401 tablet.cc:1722] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:48.637619 7401 tablet.cc:1620] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:48.640739 1671 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 3f91e3e73cf34ecbb80faab9025cb115 (table rebalancer_dynamic_table [id=32579d5f9fd04dd08d2d5f17cd9ef5bf]). Expected version 1 got 0
I20240503 08:12:48.650995 1671 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=32579d5f9fd04dd08d2d5f17cd9ef5bf] alter complete (version 1)
I20240503 08:12:48.651973 7399 tablet.cc:1722] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:48.655768 1671 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet 3f91e3e73cf34ecbb80faab9025cb115 (table rebalancer_dynamic_table [id=32579d5f9fd04dd08d2d5f17cd9ef5bf]). Expected version 1 got 0
I20240503 08:12:48.659994 7398 tablet.cc:1722] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:48.660722 7398 tablet.cc:1620] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:48.661291 7399 tablet.cc:1620] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:48.670550 6375 consensus_queue.cc:1035] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.677831 7336 consensus_queue.cc:1035] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.683768 6375 consensus_queue.cc:1035] T 4e09aca7245948c697579899ad5bfd96 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.726881 7241 consensus_queue.cc:1035] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:48.760887 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:48.772487 7398 tablet.cc:1722] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:48.773041 7398 tablet.cc:1620] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:48.782164 7401 tablet.cc:1722] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:48.782752 7401 tablet.cc:1620] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:48.785269 7399 tablet.cc:1722] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:48.785759 7399 tablet.cc:1620] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:48.798110 1671 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=32579d5f9fd04dd08d2d5f17cd9ef5bf] alter complete (version 2)
I20240503 08:12:48.859701 7434 ts_tablet_manager.cc:902] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:48.861358 7434 tablet_copy_client.cc:308] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:12:48.862202 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet 1d4f6c5975f8446a9f6ba8cae137bda0 from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:55553)
I20240503 08:12:48.862449 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet 1d4f6c5975f8446a9f6ba8cae137bda0 from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:55553: session id = 184c543cb05d49cc98ba1c56617be7a5-1d4f6c5975f8446a9f6ba8cae137bda0
I20240503 08:12:48.865097 4855 tablet_copy_source_session.cc:205] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:48.866401 7434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4f6c5975f8446a9f6ba8cae137bda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:48.872488 7434 tablet_copy_client.cc:783] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:48.872788 7434 tablet_copy_client.cc:647] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:48.874467 7434 tablet_copy_client.cc:523] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:48.878367 7434 tablet_bootstrap.cc:492] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:48.881049 6624 raft_consensus.cc:1050] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:12:48.881887 6624 consensus_queue.cc:237] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:48.896268 2073 raft_consensus.cc:1261] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:48.897260 7366 consensus_queue.cc:1035] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:48.899219 7434 tablet_bootstrap.cc:492] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:48.899739 7434 tablet_bootstrap.cc:492] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:12:48.900040 7434 ts_tablet_manager.cc:1367] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.012s
I20240503 08:12:48.900498 2202 raft_consensus.cc:1261] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:48.900980 7434 raft_consensus.cc:348] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:48.901243 7434 raft_consensus.cc:729] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:12:48.901551 7434 consensus_queue.cc:260] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:48.902199 7434 ts_tablet_manager.cc:1398] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:48.904413 1671 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:48.904659 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-1d4f6c5975f8446a9f6ba8cae137bda0 received from {username='slave'} at 127.27.157.196:55553
I20240503 08:12:48.904789 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-1d4f6c5975f8446a9f6ba8cae137bda0 on tablet 1d4f6c5975f8446a9f6ba8cae137bda0 with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:48.904886 7366 consensus_queue.cc:1035] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:48.907166 4835 raft_consensus.cc:1261] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:48.915498 7398 tablet.cc:1722] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:48.924721 7399 tablet.cc:1722] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:48.925415 7401 tablet.cc:1722] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:48.926538 6624 consensus_queue.cc:1035] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:48.927711 7366 raft_consensus.cc:2932] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:48.929143 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=32579d5f9fd04dd08d2d5f17cd9ef5bf] alter complete (version 3)
I20240503 08:12:48.929250 2202 raft_consensus.cc:2932] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:48.933662 4834 raft_consensus.cc:2932] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:48.937726 1668 catalog_manager.cc:5484] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:48.943436 2073 raft_consensus.cc:2932] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:48.958459 1810 consensus_queue.cc:237] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:48.960811 4834 raft_consensus.cc:1261] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:48.961469 7364 consensus_queue.cc:1035] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:48.962805 2073 raft_consensus.cc:1261] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:48.963639 7364 consensus_queue.cc:1035] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:48.964816 7364 raft_consensus.cc:2932] T fc4d5ddaaa27463cba31eed19c4c1be0 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:48.968644 2073 raft_consensus.cc:2932] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:48.969200 4834 raft_consensus.cc:2932] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:48.976274 1668 catalog_manager.cc:5484] T fc4d5ddaaa27463cba31eed19c4c1be0 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:48.980816 7456 ts_tablet_manager.cc:902] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:48.989760 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet fc4d5ddaaa27463cba31eed19c4c1be0 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:48.995057 7455 ts_tablet_manager.cc:902] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:48.996357 7456 tablet_copy_client.cc:308] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:48.997179 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 9d11cd50b562408f9d3bc24a974bdba8 from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:43591)
I20240503 08:12:48.997340 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 9d11cd50b562408f9d3bc24a974bdba8 from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:43591: session id = 00b61625d8a7460892c118a2570afda5-9d11cd50b562408f9d3bc24a974bdba8
I20240503 08:12:49.001114 1964 tablet_copy_source_session.cc:205] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:49.003830 7456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d11cd50b562408f9d3bc24a974bdba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.004307 7461 tablet_replica.cc:330] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:49.004703 7461 raft_consensus.cc:2229] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:49.005031 7461 raft_consensus.cc:2258] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:49.006556 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc4d5ddaaa27463cba31eed19c4c1be0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:49.006740 7461 ts_tablet_manager.cc:1875] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:49.017292 7461 ts_tablet_manager.cc:1888] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:49.017589 7461 log.cc:1196] T fc4d5ddaaa27463cba31eed19c4c1be0 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/fc4d5ddaaa27463cba31eed19c4c1be0
I20240503 08:12:49.018826 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet fc4d5ddaaa27463cba31eed19c4c1be0 (table rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]) successfully deleted
I20240503 08:12:49.025447 7455 tablet_copy_client.cc:308] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:49.026398 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 674e558bf5db4f169f8cdcb08c40f5fb from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:32893)
I20240503 08:12:49.026602 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 674e558bf5db4f169f8cdcb08c40f5fb from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:32893: session id = 5f95b05d111e4090bbc59dec394ed0ab-674e558bf5db4f169f8cdcb08c40f5fb
I20240503 08:12:49.030630 1964 tablet_copy_source_session.cc:205] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:49.033783 7455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674e558bf5db4f169f8cdcb08c40f5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.037992 7456 tablet_copy_client.cc:783] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:49.038386 7456 tablet_copy_client.cc:647] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:49.041138 7456 tablet_copy_client.cc:523] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:49.044160 7455 tablet_copy_client.cc:783] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:49.044543 7455 tablet_copy_client.cc:647] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:49.046352 7455 tablet_copy_client.cc:523] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:49.046859 7456 tablet_bootstrap.cc:492] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:49.052106 1671 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:49.052336 1671 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:49.057520 7455 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:49.057744 1671 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 3f91e3e73cf34ecbb80faab9025cb115
I20240503 08:12:49.058010 7066 consensus_queue.cc:1035] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.059243 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f91e3e73cf34ecbb80faab9025cb115 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:49 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:49.059721 4812 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f91e3e73cf34ecbb80faab9025cb115 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:49 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:49.061172 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f91e3e73cf34ecbb80faab9025cb115 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:49 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:49.069976 7474 tablet_replica.cc:330] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:49.070870 7474 raft_consensus.cc:2229] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:49.071141 7474 raft_consensus.cc:2258] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:49.077106 7474 ts_tablet_manager.cc:1875] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:49.081543 7109 raft_consensus.cc:1050] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:12:49.082463 7109 consensus_queue.cc:237] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:49.097179 7472 tablet_replica.cc:330] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:49.097589 7472 raft_consensus.cc:2229] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:49.100705 1944 raft_consensus.cc:1261] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:49.100754 7472 raft_consensus.cc:2258] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:49.101387 7055 consensus_queue.cc:1035] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:49.101755 1671 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:49.102296 7472 ts_tablet_manager.cc:1875] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:49.102809 7471 tablet_replica.cc:330] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:49.104851 7474 ts_tablet_manager.cc:1888] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:49.105145 7474 log.cc:1196] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/3f91e3e73cf34ecbb80faab9025cb115
I20240503 08:12:49.105789 7474 ts_tablet_manager.cc:1909] T 3f91e3e73cf34ecbb80faab9025cb115 P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:49.110244 4834 raft_consensus.cc:1261] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:49.110411 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 3f91e3e73cf34ecbb80faab9025cb115 (table rebalancer_dynamic_table_new_name [id=32579d5f9fd04dd08d2d5f17cd9ef5bf]) successfully deleted
I20240503 08:12:49.111608 7055 consensus_queue.cc:1035] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:49.118808 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 47dde43816964e89bfb235aad34e9583 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.119505 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dde43816964e89bfb235aad34e9583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.119877 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 826e7d6e6d2947fbb2a178e8e2a32d56 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.120317 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 826e7d6e6d2947fbb2a178e8e2a32d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.127249 4681 tablet_service.cc:1467] Processing CreateTablet for tablet 47dde43816964e89bfb235aad34e9583 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.127362 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 826e7d6e6d2947fbb2a178e8e2a32d56 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.127777 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dde43816964e89bfb235aad34e9583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.128106 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 826e7d6e6d2947fbb2a178e8e2a32d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.131774 7472 ts_tablet_manager.cc:1888] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:49.132673 7488 tablet_bootstrap.cc:492] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:49.134336 7488 tablet_bootstrap.cc:654] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.134634 7472 log.cc:1196] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/3f91e3e73cf34ecbb80faab9025cb115
I20240503 08:12:49.135517 7472 ts_tablet_manager.cc:1909] T 3f91e3e73cf34ecbb80faab9025cb115 P f24648c92c384f1ba349eee801f985e2: Deleting consensus metadata
I20240503 08:12:49.114105 7066 consensus_queue.cc:1035] T c4e697bc477a4899b69e4fdcc4e235e7 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.124231 4549 tablet_service.cc:1467] Processing CreateTablet for tablet 53ce7a6567274050a2b377a9c650ccc4 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.138883 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ce7a6567274050a2b377a9c650ccc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.142174 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 53ce7a6567274050a2b377a9c650ccc4 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.124682 4547 tablet_service.cc:1467] Processing CreateTablet for tablet 47dde43816964e89bfb235aad34e9583 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.143090 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dde43816964e89bfb235aad34e9583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.153633 7488 tablet_bootstrap.cc:492] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:49.153913 7488 ts_tablet_manager.cc:1367] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240503 08:12:49.154721 7488 raft_consensus.cc:348] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:49.155069 7488 raft_consensus.cc:374] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.155189 7488 raft_consensus.cc:729] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.155519 7488 consensus_queue.cc:260] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:49.156185 7488 ts_tablet_manager.cc:1398] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:49.157507 7489 tablet_bootstrap.cc:492] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:49.158943 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ce7a6567274050a2b377a9c650ccc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.161928 1671 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:49.164842 7184 raft_consensus.cc:1050] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:49.169513 7184 consensus_queue.cc:237] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:49.179958 2052 tablet_service.cc:1467] Processing CreateTablet for tablet bb087a8997c24f139e6464bb5a2cae2b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da256aa0623f4e4bac8758911825292c]), partition=
I20240503 08:12:49.180408 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb087a8997c24f139e6464bb5a2cae2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.181218 4946 tablet_service.cc:1467] Processing CreateTablet for tablet bb087a8997c24f139e6464bb5a2cae2b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da256aa0623f4e4bac8758911825292c]), partition=
I20240503 08:12:49.181607 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb087a8997c24f139e6464bb5a2cae2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.182695 7489 tablet_bootstrap.cc:654] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.184885 7493 tablet_bootstrap.cc:492] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:49.187070 7493 tablet_bootstrap.cc:654] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.198089 7493 tablet_bootstrap.cc:492] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:49.198812 7455 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:49.199512 7455 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:49.200269 7455 ts_tablet_manager.cc:1367] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.143s user 0.044s sys 0.012s
I20240503 08:12:49.201366 7493 ts_tablet_manager.cc:1367] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240503 08:12:49.201511 7455 raft_consensus.cc:348] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:49.201884 7455 raft_consensus.cc:729] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:49.202204 7455 consensus_queue.cc:260] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:49.202639 7493 raft_consensus.cc:348] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:49.204059 7493 raft_consensus.cc:374] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.204638 7455 ts_tablet_manager.cc:1398] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:49.204555 7493 raft_consensus.cc:729] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.205138 7493 consensus_queue.cc:260] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:49.205363 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-674e558bf5db4f169f8cdcb08c40f5fb received from {username='slave'} at 127.27.157.197:32893
I20240503 08:12:49.205482 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-674e558bf5db4f169f8cdcb08c40f5fb on tablet 674e558bf5db4f169f8cdcb08c40f5fb with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:49.229141 2202 raft_consensus.cc:1261] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:49.231432 7499 consensus_queue.cc:1035] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:49.232182 7493 ts_tablet_manager.cc:1398] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20240503 08:12:49.235136 7493 tablet_bootstrap.cc:492] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:49.237576 4548 tablet_service.cc:1467] Processing CreateTablet for tablet bb087a8997c24f139e6464bb5a2cae2b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da256aa0623f4e4bac8758911825292c]), partition=
I20240503 08:12:49.238041 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb087a8997c24f139e6464bb5a2cae2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.243899 7471 raft_consensus.cc:2229] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:49.245127 7471 raft_consensus.cc:2258] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:49.252548 7493 tablet_bootstrap.cc:654] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.253017 7505 tablet_bootstrap.cc:492] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:49.253674 7471 ts_tablet_manager.cc:1875] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:49.255847 7505 tablet_bootstrap.cc:654] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
W20240503 08:12:49.256024 4637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.193:41181, user_credentials={real_user=slave}} blocked reactor thread for 99524.5us
I20240503 08:12:49.256912 7498 tablet_bootstrap.cc:492] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:49.259661 7498 tablet_bootstrap.cc:654] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.262219 7500 raft_consensus.cc:2932] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:49.263224 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 53ce7a6567274050a2b377a9c650ccc4 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.263823 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ce7a6567274050a2b377a9c650ccc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:49.264153 4568 raft_consensus.cc:1261] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:49.265875 2202 raft_consensus.cc:2932] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:49.263253 4814 tablet_service.cc:1467] Processing CreateTablet for tablet 826e7d6e6d2947fbb2a178e8e2a32d56 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:49.267994 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 826e7d6e6d2947fbb2a178e8e2a32d56. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:12:49.270483 2010 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.199:39455, user_credentials={real_user=slave}} blocked reactor thread for 159387us
I20240503 08:12:49.265416 7186 consensus_queue.cc:1035] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 08:12:49.286808 4767 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.197:42465, user_credentials={real_user=slave}} blocked reactor thread for 215167us
I20240503 08:12:49.295372 7489 tablet_bootstrap.cc:492] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:49.297318 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet 3f91e3e73cf34ecbb80faab9025cb115 (table rebalancer_dynamic_table_new_name [id=32579d5f9fd04dd08d2d5f17cd9ef5bf]) successfully deleted
I20240503 08:12:49.297407 7515 tablet_bootstrap.cc:492] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:49.298133 7489 ts_tablet_manager.cc:1367] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.141s user 0.011s sys 0.006s
I20240503 08:12:49.299258 7505 tablet_bootstrap.cc:492] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:49.300310 7489 raft_consensus.cc:348] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:49.300976 7505 ts_tablet_manager.cc:1367] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.001s
I20240503 08:12:49.301976 7505 raft_consensus.cc:348] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:49.307426 7489 raft_consensus.cc:374] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.307680 7489 raft_consensus.cc:729] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.308104 7505 raft_consensus.cc:374] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.308353 7489 consensus_queue.cc:260] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:49.308703 7505 raft_consensus.cc:729] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.309233 1671 catalog_manager.cc:5484] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:49.309947 7505 consensus_queue.cc:260] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:49.310686 7489 ts_tablet_manager.cc:1398] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 08:12:49.311270 7515 tablet_bootstrap.cc:654] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
W20240503 08:12:49.312783 4902 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.194:41975, user_credentials={real_user=slave}} blocked reactor thread for 44538.6us
I20240503 08:12:49.316345 2202 raft_consensus.cc:1261] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:49.316231 1944 raft_consensus.cc:2932] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:49.317826 7509 consensus_queue.cc:1035] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:49.313156 4834 raft_consensus.cc:2932] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:49.329265 7505 ts_tablet_manager.cc:1398] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20240503 08:12:49.329502 7493 tablet_bootstrap.cc:492] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:49.329756 7493 ts_tablet_manager.cc:1367] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.095s user 0.006s sys 0.000s
I20240503 08:12:49.330086 7489 tablet_bootstrap.cc:492] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:49.330986 7493 raft_consensus.cc:348] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:49.331298 7493 raft_consensus.cc:374] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.331456 7493 raft_consensus.cc:729] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.331811 7493 consensus_queue.cc:260] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:49.332082 7498 tablet_bootstrap.cc:492] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:49.332285 7498 ts_tablet_manager.cc:1367] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.000s
I20240503 08:12:49.332612 7493 ts_tablet_manager.cc:1398] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:49.333163 7498 raft_consensus.cc:348] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:49.333482 7498 raft_consensus.cc:374] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.333634 7498 raft_consensus.cc:729] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.333930 7498 consensus_queue.cc:260] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:49.334596 7498 ts_tablet_manager.cc:1398] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:49.342592 7471 ts_tablet_manager.cc:1888] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:49.344599 7515 tablet_bootstrap.cc:492] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:49.344854 7471 log.cc:1196] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/3f91e3e73cf34ecbb80faab9025cb115
I20240503 08:12:49.346103 7471 ts_tablet_manager.cc:1909] T 3f91e3e73cf34ecbb80faab9025cb115 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:49.346091 7515 ts_tablet_manager.cc:1367] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.006s
I20240503 08:12:49.347388 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 3f91e3e73cf34ecbb80faab9025cb115 (table rebalancer_dynamic_table_new_name [id=32579d5f9fd04dd08d2d5f17cd9ef5bf]) successfully deleted
I20240503 08:12:49.348453 7505 tablet_bootstrap.cc:492] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:49.349052 7515 raft_consensus.cc:348] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:49.349303 7515 raft_consensus.cc:374] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.350746 7505 tablet_bootstrap.cc:654] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.351728 7515 raft_consensus.cc:729] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.352443 7515 consensus_queue.cc:260] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:49.353039 7489 tablet_bootstrap.cc:654] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.353132 7515 ts_tablet_manager.cc:1398] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 08:12:49.353440 7515 tablet_bootstrap.cc:492] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:49.361266 7510 raft_consensus.cc:2932] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:49.362337 7505 tablet_bootstrap.cc:492] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:49.362520 7505 ts_tablet_manager.cc:1367] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240503 08:12:49.363480 7505 raft_consensus.cc:348] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:49.363720 7505 raft_consensus.cc:374] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.363790 7505 raft_consensus.cc:729] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.364045 7505 consensus_queue.cc:260] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:49.364591 7505 ts_tablet_manager.cc:1398] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:49.364945 7505 tablet_bootstrap.cc:492] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:49.376490 7505 tablet_bootstrap.cc:654] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.355288 7515 tablet_bootstrap.cc:654] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.380136 4568 raft_consensus.cc:2932] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:49.386355 7515 tablet_bootstrap.cc:492] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:49.391137 7515 ts_tablet_manager.cc:1367] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20240503 08:12:49.391427 7505 tablet_bootstrap.cc:492] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:49.391737 7505 ts_tablet_manager.cc:1367] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240503 08:12:49.392930 7515 raft_consensus.cc:348] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:49.402336 7505 raft_consensus.cc:348] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:49.402757 7505 raft_consensus.cc:374] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.402930 7505 raft_consensus.cc:729] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.403349 7505 consensus_queue.cc:260] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:49.404256 7505 ts_tablet_manager.cc:1398] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:12:49.407168 4701 raft_consensus.cc:1261] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:49.408895 7511 consensus_queue.cc:1035] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:49.409157 1668 catalog_manager.cc:5484] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:49.414214 7515 raft_consensus.cc:374] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.414415 7515 raft_consensus.cc:729] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.414773 7515 consensus_queue.cc:260] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:49.415680 7515 ts_tablet_manager.cc:1398] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.024s user 0.001s sys 0.002s
I20240503 08:12:49.417192 2204 raft_consensus.cc:2932] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:49.418191 2202 raft_consensus.cc:1203] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:49.437592 4701 raft_consensus.cc:2932] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:49.453868 7489 tablet_bootstrap.cc:492] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:49.454171 7489 ts_tablet_manager.cc:1367] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.124s user 0.034s sys 0.037s
I20240503 08:12:49.455132 7489 raft_consensus.cc:348] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:49.455577 7489 raft_consensus.cc:374] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.455739 7489 raft_consensus.cc:729] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.456076 7489 consensus_queue.cc:260] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:49.456801 7489 ts_tablet_manager.cc:1398] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:49.459025 7489 tablet_bootstrap.cc:492] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:49.461140 7489 tablet_bootstrap.cc:654] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:49.474936 7489 tablet_bootstrap.cc:492] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:49.475219 7489 ts_tablet_manager.cc:1367] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20240503 08:12:49.476105 7489 raft_consensus.cc:348] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:49.476433 7489 raft_consensus.cc:374] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:49.476651 7489 raft_consensus.cc:729] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:49.477062 7489 consensus_queue.cc:260] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:49.480685 7502 raft_consensus.cc:482] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.481009 7502 raft_consensus.cc:504] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:49.482957 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ce7a6567274050a2b377a9c650ccc4" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:49.483294 4567 raft_consensus.cc:2454] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:49.484134 4635 leader_election.cc:302] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:49.484520 7179 raft_consensus.cc:2781] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.484630 7179 raft_consensus.cc:482] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.484730 7179 raft_consensus.cc:3037] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.488734 7179 raft_consensus.cc:504] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:49.490379 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ce7a6567274050a2b377a9c650ccc4" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:49.490658 4834 raft_consensus.cc:3037] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.494246 4834 raft_consensus.cc:2454] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:49.494922 4638 leader_election.cc:302] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:49.495368 7492 raft_consensus.cc:2781] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.495520 7492 raft_consensus.cc:686] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:49.495898 7492 consensus_queue.cc:237] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:49.498525 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ce7a6567274050a2b377a9c650ccc4" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:49.498654 4568 raft_consensus.cc:1203] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:49.498768 4567 raft_consensus.cc:3037] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.501658 4567 raft_consensus.cc:2454] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:49.502395 7179 leader_election.cc:288] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:49.505340 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ce7a6567274050a2b377a9c650ccc4" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:49.505762 4834 raft_consensus.cc:2362] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1. Re-sending same reply.
I20240503 08:12:49.506361 7502 leader_election.cc:288] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:49.506167 1668 catalog_manager.cc:5484] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:49.507175 7489 ts_tablet_manager.cc:1398] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20240503 08:12:49.518291 7456 tablet_bootstrap.cc:492] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:49.521854 7456 tablet_bootstrap.cc:492] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:12:49.523013 7456 ts_tablet_manager.cc:1367] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.476s user 0.047s sys 0.009s
I20240503 08:12:49.524190 7456 raft_consensus.cc:348] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:49.525944 7456 raft_consensus.cc:729] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:12:49.526407 7456 consensus_queue.cc:260] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:49.528601 7376 raft_consensus.cc:482] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.528798 7376 raft_consensus.cc:504] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:49.530143 7376 leader_election.cc:288] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:49.530903 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826e7d6e6d2947fbb2a178e8e2a32d56" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:49.531142 4701 raft_consensus.cc:2454] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:49.531098 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826e7d6e6d2947fbb2a178e8e2a32d56" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:49.531356 2072 raft_consensus.cc:2454] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:49.531682 4770 leader_election.cc:302] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:49.532248 7376 raft_consensus.cc:2781] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.532366 7376 raft_consensus.cc:482] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.532392 7456 ts_tablet_manager.cc:1398] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.009s user 0.000s sys 0.004s
I20240503 08:12:49.532455 7376 raft_consensus.cc:3037] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.533079 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-9d11cd50b562408f9d3bc24a974bdba8 received from {username='slave'} at 127.27.157.200:43591
I20240503 08:12:49.533181 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 00b61625d8a7460892c118a2570afda5-9d11cd50b562408f9d3bc24a974bdba8 on tablet 9d11cd50b562408f9d3bc24a974bdba8 with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:12:49.535112 7376 raft_consensus.cc:504] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:49.536895 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826e7d6e6d2947fbb2a178e8e2a32d56" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:49.537160 4701 raft_consensus.cc:3037] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.540058 7376 leader_election.cc:288] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:49.540568 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "826e7d6e6d2947fbb2a178e8e2a32d56" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:49.540793 2072 raft_consensus.cc:3037] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.543751 2072 raft_consensus.cc:2454] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:49.544333 4768 leader_election.cc:302] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:49.544790 7376 raft_consensus.cc:2781] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.544945 7376 raft_consensus.cc:686] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:49.545267 7376 consensus_queue.cc:237] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:49.563269 4701 raft_consensus.cc:2454] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:49.560484 1668 catalog_manager.cc:5484] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:49.579638 7511 raft_consensus.cc:482] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.579852 7511 raft_consensus.cc:504] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:49.580663 7511 leader_election.cc:288] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:49.581487 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb087a8997c24f139e6464bb5a2cae2b" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:49.581730 2072 raft_consensus.cc:2454] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:49.582309 4901 leader_election.cc:302] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:49.582736 7511 raft_consensus.cc:2781] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.582732 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb087a8997c24f139e6464bb5a2cae2b" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:49.582832 7511 raft_consensus.cc:482] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.582928 7511 raft_consensus.cc:3037] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.582993 4567 raft_consensus.cc:2454] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:49.586464 7511 raft_consensus.cc:504] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:49.588977 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb087a8997c24f139e6464bb5a2cae2b" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:49.589133 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb087a8997c24f139e6464bb5a2cae2b" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:49.589251 2072 raft_consensus.cc:3037] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.589375 4567 raft_consensus.cc:3037] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.592268 2072 raft_consensus.cc:2454] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:49.592268 4567 raft_consensus.cc:2454] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:49.592828 4901 leader_election.cc:302] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 43e2e01cdd314f9c872c4336ef74250a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:49.593017 7511 leader_election.cc:288] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:49.593197 7184 raft_consensus.cc:2781] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.593333 7184 raft_consensus.cc:686] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:49.593772 7184 consensus_queue.cc:237] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:49.597960 1668 catalog_manager.cc:5484] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:49.599388 7336 raft_consensus.cc:482] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:49.599535 7336 raft_consensus.cc:504] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:49.600275 7336 leader_election.cc:288] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:49.603035 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dde43816964e89bfb235aad34e9583" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:49.603370 4701 raft_consensus.cc:2454] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:49.604629 1880 leader_election.cc:302] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:49.604921 7336 raft_consensus.cc:2781] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:49.605007 7336 raft_consensus.cc:482] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:49.605067 7336 raft_consensus.cc:3037] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.605530 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dde43816964e89bfb235aad34e9583" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:49.605787 4568 raft_consensus.cc:2454] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:49.607865 7336 raft_consensus.cc:504] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:49.608489 7336 leader_election.cc:288] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:49.609879 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dde43816964e89bfb235aad34e9583" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:49.610085 4703 raft_consensus.cc:3037] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.610572 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47dde43816964e89bfb235aad34e9583" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:49.610832 4568 raft_consensus.cc:3037] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:49.613070 4703 raft_consensus.cc:2454] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:49.613292 4568 raft_consensus.cc:2454] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:49.613693 1880 leader_election.cc:302] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:49.614070 7336 raft_consensus.cc:2781] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:49.614320 7336 raft_consensus.cc:686] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:49.614650 7336 consensus_queue.cc:237] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:49.618778 1665 catalog_manager.cc:5484] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:49.731084 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:49.749598 4567 raft_consensus.cc:1261] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:49.749817 2072 raft_consensus.cc:1261] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:49.750777 7511 consensus_queue.cc:1035] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.751345 7184 consensus_queue.cc:1035] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.759964 7579 tablet.cc:1722] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:49.760789 7579 tablet.cc:1620] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:49.771607 1665 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet bb087a8997c24f139e6464bb5a2cae2b (table rebalancer_dynamic_table [id=da256aa0623f4e4bac8758911825292c]). Expected version 1 got 0
I20240503 08:12:49.774080 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=da256aa0623f4e4bac8758911825292c] alter complete (version 1)
I20240503 08:12:49.778179 7580 tablet.cc:1722] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:49.779039 7580 tablet.cc:1620] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:49.788319 7582 tablet.cc:1722] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:49.788959 7582 tablet.cc:1620] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: MemRowSet was empty: no flush needed.
I20240503 08:12:49.789732 1665 catalog_manager.cc:5576] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117) does not have the latest schema for tablet bb087a8997c24f139e6464bb5a2cae2b (table rebalancer_dynamic_table [id=da256aa0623f4e4bac8758911825292c]). Expected version 1 got 0
I20240503 08:12:49.801523 2072 consensus_queue.cc:237] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:49.803589 4834 raft_consensus.cc:1261] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:49.803748 2204 raft_consensus.cc:1261] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:49.804188 7109 consensus_queue.cc:1035] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:49.804522 7499 consensus_queue.cc:1035] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:49.806839 7500 raft_consensus.cc:2932] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:49.807652 4834 raft_consensus.cc:2932] T b00417d7369a46aabfa933109c645c2c P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:49.807601 2204 raft_consensus.cc:2932] T b00417d7369a46aabfa933109c645c2c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:49.811159 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b00417d7369a46aabfa933109c645c2c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:49.814195 1672 catalog_manager.cc:5484] T b00417d7369a46aabfa933109c645c2c P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 5 to 6, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:49.818193 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet b00417d7369a46aabfa933109c645c2c with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:49.818856 7583 tablet_replica.cc:330] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:49.819144 7583 raft_consensus.cc:2229] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:49.819442 7583 raft_consensus.cc:2258] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:49.820367 7583 ts_tablet_manager.cc:1875] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:49.825402 7583 ts_tablet_manager.cc:1888] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:49.825520 7583 log.cc:1196] T b00417d7369a46aabfa933109c645c2c P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/b00417d7369a46aabfa933109c645c2c
I20240503 08:12:49.826275 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet b00417d7369a46aabfa933109c645c2c (table rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]) successfully deleted
I20240503 08:12:49.880554 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:49.888156 7579 tablet.cc:1722] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:49.888803 7580 tablet.cc:1722] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:49.889045 7582 tablet.cc:1722] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:49.889293 7580 tablet.cc:1620] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:49.889333 7579 tablet.cc:1620] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:49.889674 7582 tablet.cc:1620] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: MemRowSet was empty: no flush needed.
I20240503 08:12:49.900110 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=da256aa0623f4e4bac8758911825292c] alter complete (version 2)
I20240503 08:12:49.926441 4966 raft_consensus.cc:1203] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 08:12:49.941802 4966 consensus_queue.cc:237] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:49.943939 4701 raft_consensus.cc:1261] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:49.944725 7186 consensus_queue.cc:1035] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:49.946566 4567 raft_consensus.cc:1261] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:49.947582 7184 raft_consensus.cc:2932] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:49.948268 4701 raft_consensus.cc:2932] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:49.950989 7186 consensus_queue.cc:1035] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:49.954283 4567 raft_consensus.cc:2932] T 8737ef0936c641ccaf0b804b1d9c3e07 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:49.956300 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8737ef0936c641ccaf0b804b1d9c3e07 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:49.959208 1672 catalog_manager.cc:5484] T 8737ef0936c641ccaf0b804b1d9c3e07 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 3 to 4, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:49.965309 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 8737ef0936c641ccaf0b804b1d9c3e07 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:49.966149 7590 tablet_replica.cc:330] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:49.966212 7179 consensus_queue.cc:1035] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.966797 7590 raft_consensus.cc:2229] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:49.967486 7590 raft_consensus.cc:2258] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:49.969833 7590 ts_tablet_manager.cc:1875] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:49.976531 7376 raft_consensus.cc:1050] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:12:49.977020 7590 ts_tablet_manager.cc:1888] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:49.977164 7590 log.cc:1196] T 8737ef0936c641ccaf0b804b1d9c3e07 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/8737ef0936c641ccaf0b804b1d9c3e07
I20240503 08:12:49.977365 7376 consensus_queue.cc:237] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:49.978703 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 8737ef0936c641ccaf0b804b1d9c3e07 (table rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]) successfully deleted
I20240503 08:12:49.981775 7507 consensus_queue.cc:1035] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:49.986132 4701 raft_consensus.cc:1261] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:49.986717 2204 raft_consensus.cc:1261] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:49.987013 7595 consensus_queue.cc:1035] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:49.987596 7476 consensus_queue.cc:1035] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:49.990403 1944 raft_consensus.cc:1261] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:49.991351 7476 consensus_queue.cc:1035] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:50.007550 1668 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:50.021199 7579 tablet.cc:1722] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:50.022145 7580 tablet.cc:1722] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:50.024341 7582 tablet.cc:1722] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:50.033406 7595 consensus_queue.cc:1035] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.036788 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=da256aa0623f4e4bac8758911825292c] alter complete (version 3)
I20240503 08:12:50.057310 7476 consensus_queue.cc:1035] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.064967 7595 raft_consensus.cc:2932] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:50.066124 1944 raft_consensus.cc:2932] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:50.081136 2204 raft_consensus.cc:2932] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:50.087626 6375 raft_consensus.cc:1050] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:50.088459 6375 consensus_queue.cc:237] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:50.090456 1672 catalog_manager.cc:5484] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:50.092034 4703 raft_consensus.cc:2932] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:50.102583 2202 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:50.104028 4568 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:50.105423 2072 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:50.107671 6375 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20240503 08:12:50.108038 7241 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:50.108323 7336 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:50.112766 4835 consensus_queue.cc:237] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:50.119282 7336 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:50.119869 2202 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:50.124835 4701 raft_consensus.cc:1261] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:50.125622 7476 consensus_queue.cc:1035] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:50.126682 1665 catalog_manager.cc:5484] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:50.127658 2072 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:50.129887 7606 raft_consensus.cc:2932] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:50.136060 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d4f6c5975f8446a9f6ba8cae137bda0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:50.136575 4701 raft_consensus.cc:2932] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:50.140126 1664 catalog_manager.cc:5484] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 3 to 4, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:50.143362 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:50.149062 1664 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:50.149288 1664 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:50.169137 2204 raft_consensus.cc:1261] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:50.171147 7606 consensus_queue.cc:1035] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:50.178758 1664 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet bb087a8997c24f139e6464bb5a2cae2b
I20240503 08:12:50.179957 2204 raft_consensus.cc:2932] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:50.180780 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet bb087a8997c24f139e6464bb5a2cae2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:50 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:50.181197 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet bb087a8997c24f139e6464bb5a2cae2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:50 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:50.182377 4946 tablet_service.cc:1514] Processing DeleteTablet for tablet bb087a8997c24f139e6464bb5a2cae2b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:50 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:50.198544 4945 tablet_service.cc:1467] Processing CreateTablet for tablet c6994370a532472893cc934337f3cb6e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.198647 4547 tablet_service.cc:1467] Processing CreateTablet for tablet c6994370a532472893cc934337f3cb6e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.199242 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6994370a532472893cc934337f3cb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.199292 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6994370a532472893cc934337f3cb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.212177 7621 tablet_replica.cc:330] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:50.212424 7621 raft_consensus.cc:2229] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.212646 7621 raft_consensus.cc:2258] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.213497 7621 ts_tablet_manager.cc:1875] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:50.213888 7622 tablet_replica.cc:330] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:50.214167 7622 raft_consensus.cc:2229] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 08:12:50.214120 1877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.197:42465, user_credentials={real_user=slave}} blocked reactor thread for 84749.9us
I20240503 08:12:50.214437 7622 raft_consensus.cc:2258] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.214859 4568 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:50.215981 7622 ts_tablet_manager.cc:1875] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:50.216845 7623 tablet_replica.cc:330] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:50.216897 4814 tablet_service.cc:1467] Processing CreateTablet for tablet c6994370a532472893cc934337f3cb6e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.217135 7623 raft_consensus.cc:2229] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:50.217412 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6994370a532472893cc934337f3cb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.218900 7623 raft_consensus.cc:2258] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.221467 7623 ts_tablet_manager.cc:1875] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:50.222148 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d4f6c5975f8446a9f6ba8cae137bda0 with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:50.222466 7583 tablet_replica.cc:330] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:50.222628 7583 raft_consensus.cc:2229] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.222841 7583 raft_consensus.cc:2258] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.223800 7583 ts_tablet_manager.cc:1875] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:50.224357 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 98ad826dca7c4343afcae34bf1a40da8 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.224834 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ad826dca7c4343afcae34bf1a40da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.230486 7583 ts_tablet_manager.cc:1888] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:50.230624 7583 log.cc:1196] T 1d4f6c5975f8446a9f6ba8cae137bda0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/1d4f6c5975f8446a9f6ba8cae137bda0
I20240503 08:12:50.232748 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 1d4f6c5975f8446a9f6ba8cae137bda0 (table rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]) successfully deleted
I20240503 08:12:50.233984 7617 consensus_queue.cc:1035] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.246057 1944 consensus_queue.cc:237] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:50.249261 2073 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:50.250345 7618 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:50.254204 4567 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:50.254460 7617 consensus_queue.cc:1035] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.255188 7241 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:50.257259 4567 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:50.253444 7616 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:50.266582 2073 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:50.270985 7622 ts_tablet_manager.cc:1888] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:50.271521 7629 tablet_bootstrap.cc:492] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:50.272575 7622 log.cc:1196] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb087a8997c24f139e6464bb5a2cae2b
I20240503 08:12:50.273226 7621 ts_tablet_manager.cc:1888] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:50.274330 7621 log.cc:1196] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/bb087a8997c24f139e6464bb5a2cae2b
I20240503 08:12:50.278720 7622 ts_tablet_manager.cc:1909] T bb087a8997c24f139e6464bb5a2cae2b P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:50.283486 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:50.283993 7621 ts_tablet_manager.cc:1909] T bb087a8997c24f139e6464bb5a2cae2b P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:50.285351 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0bffb0aa304bd184b8ad26bc94fd18 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.285421 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 98ad826dca7c4343afcae34bf1a40da8 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.285491 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet bb087a8997c24f139e6464bb5a2cae2b (table rebalancer_dynamic_table_new_name [id=da256aa0623f4e4bac8758911825292c]) successfully deleted
I20240503 08:12:50.285813 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0bffb0aa304bd184b8ad26bc94fd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.286356 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 98ad826dca7c4343afcae34bf1a40da8 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.286525 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0bffb0aa304bd184b8ad26bc94fd18 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.286785 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ad826dca7c4343afcae34bf1a40da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.286968 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0bffb0aa304bd184b8ad26bc94fd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.287513 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ad826dca7c4343afcae34bf1a40da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.292310 7629 tablet_bootstrap.cc:654] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.295969 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0bffb0aa304bd184b8ad26bc94fd18 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:50.296998 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0bffb0aa304bd184b8ad26bc94fd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.306708 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet bb087a8997c24f139e6464bb5a2cae2b (table rebalancer_dynamic_table_new_name [id=da256aa0623f4e4bac8758911825292c]) successfully deleted
I20240503 08:12:50.312080 7241 raft_consensus.cc:1050] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:12:50.313130 7241 consensus_queue.cc:237] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:50.320160 1668 catalog_manager.cc:5484] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:50.326164 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 674e558bf5db4f169f8cdcb08c40f5fb with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:50.326575 7590 tablet_replica.cc:330] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:50.326790 7590 raft_consensus.cc:2229] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.326996 7590 raft_consensus.cc:2258] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:50.328137 7590 ts_tablet_manager.cc:1875] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:50.332118 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 674e558bf5db4f169f8cdcb08c40f5fb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:50.296411 7623 ts_tablet_manager.cc:1888] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:50.336787 7623 log.cc:1196] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/bb087a8997c24f139e6464bb5a2cae2b
I20240503 08:12:50.337498 7623 ts_tablet_manager.cc:1909] T bb087a8997c24f139e6464bb5a2cae2b P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:50.339283 4947 tablet_service.cc:1467] Processing CreateTablet for tablet c11dc200302b4d64bcdceb32e5171048 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8a49b8cedbec4f2a8efa8bb7c5a902cd]), partition=
I20240503 08:12:50.339630 1923 tablet_service.cc:1467] Processing CreateTablet for tablet c11dc200302b4d64bcdceb32e5171048 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8a49b8cedbec4f2a8efa8bb7c5a902cd]), partition=
I20240503 08:12:50.339701 7590 ts_tablet_manager.cc:1888] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:50.339776 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11dc200302b4d64bcdceb32e5171048. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.339875 7590 log.cc:1196] T 674e558bf5db4f169f8cdcb08c40f5fb P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/674e558bf5db4f169f8cdcb08c40f5fb
I20240503 08:12:50.339969 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11dc200302b4d64bcdceb32e5171048. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.347545 4812 tablet_service.cc:1467] Processing CreateTablet for tablet c11dc200302b4d64bcdceb32e5171048 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8a49b8cedbec4f2a8efa8bb7c5a902cd]), partition=
I20240503 08:12:50.349211 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11dc200302b4d64bcdceb32e5171048. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:50.354713 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 674e558bf5db4f169f8cdcb08c40f5fb (table rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]) successfully deleted
I20240503 08:12:50.369131 7639 tablet_bootstrap.cc:492] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:50.373209 7641 tablet_bootstrap.cc:492] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:50.382377 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet bb087a8997c24f139e6464bb5a2cae2b (table rebalancer_dynamic_table_new_name [id=da256aa0623f4e4bac8758911825292c]) successfully deleted
I20240503 08:12:50.391690 7629 tablet_bootstrap.cc:492] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:50.391867 7629 ts_tablet_manager.cc:1367] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.123s user 0.009s sys 0.014s
I20240503 08:12:50.392619 7624 tablet_bootstrap.cc:492] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:50.392894 7629 raft_consensus.cc:348] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.393294 7629 raft_consensus.cc:374] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.393467 7629 raft_consensus.cc:729] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.393836 7629 consensus_queue.cc:260] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.394644 7629 ts_tablet_manager.cc:1398] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:50.395112 7641 tablet_bootstrap.cc:654] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.395124 7629 tablet_bootstrap.cc:492] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:50.397687 7629 tablet_bootstrap.cc:654] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.398559 7639 tablet_bootstrap.cc:654] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.413771 7624 tablet_bootstrap.cc:654] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.426126 7627 tablet_bootstrap.cc:492] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:50.426402 7624 tablet_bootstrap.cc:492] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:50.426614 7624 ts_tablet_manager.cc:1367] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.003s
I20240503 08:12:50.427544 7624 raft_consensus.cc:348] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.427918 7624 raft_consensus.cc:374] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.428066 7624 raft_consensus.cc:729] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.428431 7624 consensus_queue.cc:260] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.428730 7640 tablet_bootstrap.cc:492] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:50.430526 7640 tablet_bootstrap.cc:654] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.433969 7624 ts_tablet_manager.cc:1398] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:50.435002 7624 tablet_bootstrap.cc:492] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:50.441191 7624 tablet_bootstrap.cc:654] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.445943 4966 raft_consensus.cc:1261] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:50.446509 2204 raft_consensus.cc:1261] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:50.447614 2073 raft_consensus.cc:1261] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:50.448534 7618 consensus_queue.cc:1035] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:50.448828 7616 consensus_queue.cc:1035] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:50.449373 7618 consensus_queue.cc:1035] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:50.457196 7624 tablet_bootstrap.cc:492] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:50.457465 7624 ts_tablet_manager.cc:1367] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.001s
I20240503 08:12:50.461623 7624 raft_consensus.cc:348] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.461897 7624 raft_consensus.cc:374] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.461977 7624 raft_consensus.cc:729] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.463560 7624 consensus_queue.cc:260] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.464140 7179 raft_consensus.cc:482] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.464310 7179 raft_consensus.cc:504] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.464586 7641 tablet_bootstrap.cc:492] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:50.464722 7641 ts_tablet_manager.cc:1367] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.008s
I20240503 08:12:50.464942 7179 leader_election.cc:288] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:50.465463 7641 raft_consensus.cc:348] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.465680 7641 raft_consensus.cc:374] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.465746 7641 raft_consensus.cc:729] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.466015 4566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ad826dca7c4343afcae34bf1a40da8" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:50.466920 7639 tablet_bootstrap.cc:492] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
W20240503 08:12:50.467077 4635 leader_election.cc:341] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:50.468444 7639 ts_tablet_manager.cc:1367] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.099s user 0.007s sys 0.010s
I20240503 08:12:50.468422 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ad826dca7c4343afcae34bf1a40da8" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:50.468806 7624 ts_tablet_manager.cc:1398] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 08:12:50.469465 7627 tablet_bootstrap.cc:654] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
W20240503 08:12:50.469493 4638 leader_election.cc:341] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:50.469638 4638 leader_election.cc:302] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc; no voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2
I20240503 08:12:50.469682 7639 raft_consensus.cc:348] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.469877 7639 raft_consensus.cc:374] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.469964 7639 raft_consensus.cc:729] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.470293 7502 raft_consensus.cc:2726] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:50.470261 7641 consensus_queue.cc:260] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.474128 7624 tablet_bootstrap.cc:492] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:50.474071 7618 raft_consensus.cc:2932] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:50.475208 7641 ts_tablet_manager.cc:1398] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 08:12:50.475371 7639 consensus_queue.cc:260] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.479719 7624 tablet_bootstrap.cc:654] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.480896 7639 ts_tablet_manager.cc:1398] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240503 08:12:50.481308 7639 tablet_bootstrap.cc:492] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:50.479676 2204 raft_consensus.cc:2932] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:50.483214 7639 tablet_bootstrap.cc:654] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.488788 7629 tablet_bootstrap.cc:492] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:50.489104 7629 ts_tablet_manager.cc:1367] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.094s user 0.016s sys 0.013s
I20240503 08:12:50.488445 4966 raft_consensus.cc:2932] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:50.490093 7629 raft_consensus.cc:348] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:50.490412 7629 raft_consensus.cc:374] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.490631 7629 raft_consensus.cc:729] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.491019 7629 consensus_queue.cc:260] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:50.494321 7629 ts_tablet_manager.cc:1398] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:50.496179 7624 tablet_bootstrap.cc:492] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:50.496421 7624 ts_tablet_manager.cc:1367] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240503 08:12:50.497947 7624 raft_consensus.cc:348] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:50.498302 7624 raft_consensus.cc:374] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.498450 7624 raft_consensus.cc:729] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.498771 7624 consensus_queue.cc:260] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:50.500746 2073 raft_consensus.cc:2932] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:50.510401 7624 ts_tablet_manager.cc:1398] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 08:12:50.520912 1672 catalog_manager.cc:5484] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 4 to 5, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:50.529191 7639 tablet_bootstrap.cc:492] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:50.532748 7639 ts_tablet_manager.cc:1367] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20240503 08:12:50.533821 7184 raft_consensus.cc:482] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.533998 7184 raft_consensus.cc:504] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:50.534674 7184 leader_election.cc:288] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:50.535612 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0bffb0aa304bd184b8ad26bc94fd18" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:50.535707 7639 raft_consensus.cc:348] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.536052 7639 raft_consensus.cc:374] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.536155 7639 raft_consensus.cc:729] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
W20240503 08:12:50.536366 4902 leader_election.cc:341] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:50.536777 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0bffb0aa304bd184b8ad26bc94fd18" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:50.537036 4701 raft_consensus.cc:2454] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:50.536777 7639 consensus_queue.cc:260] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.541431 4903 leader_election.cc:302] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 31f41542452b4b1eb994ff4223d039fc; no voters: b8f581390da449ccb9e299011296405e
I20240503 08:12:50.543095 7184 raft_consensus.cc:2781] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:50.543454 7184 raft_consensus.cc:482] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:50.543638 7184 raft_consensus.cc:3037] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.547148 7184 raft_consensus.cc:504] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:50.552170 7184 leader_election.cc:288] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:50.556732 7639 ts_tablet_manager.cc:1398] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240503 08:12:50.556799 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0bffb0aa304bd184b8ad26bc94fd18" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
W20240503 08:12:50.557685 4902 leader_election.cc:341] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:50.561458 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0bffb0aa304bd184b8ad26bc94fd18" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:50.561790 4703 raft_consensus.cc:3037] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.562467 1944 consensus_queue.cc:237] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:50.564111 7616 raft_consensus.cc:482] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.564252 7616 raft_consensus.cc:504] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.564859 4703 raft_consensus.cc:2454] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:50.564992 7616 leader_election.cc:288] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:50.565430 4903 leader_election.cc:302] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 31f41542452b4b1eb994ff4223d039fc; no voters: b8f581390da449ccb9e299011296405e
I20240503 08:12:50.565915 7184 raft_consensus.cc:2781] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:50.565984 7511 raft_consensus.cc:482] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.566108 7184 raft_consensus.cc:686] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:50.566159 7511 raft_consensus.cc:504] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.566419 7184 consensus_queue.cc:237] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:50.567001 7511 leader_election.cc:288] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:50.567574 4967 raft_consensus.cc:1261] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:50.568514 7336 consensus_queue.cc:1035] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:50.568666 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6994370a532472893cc934337f3cb6e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:50.568948 4568 raft_consensus.cc:2454] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:50.569519 4900 leader_election.cc:302] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:50.570942 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6994370a532472893cc934337f3cb6e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:50.571478 7184 raft_consensus.cc:2781] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:50.571564 7184 raft_consensus.cc:482] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:50.571640 7184 raft_consensus.cc:3037] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.574394 7184 raft_consensus.cc:504] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.574244 7617 raft_consensus.cc:2932] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:50.574620 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11dc200302b4d64bcdceb32e5171048" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:50.575141 7184 leader_election.cc:288] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:50.576040 2072 raft_consensus.cc:1261] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 08:12:50.577100 1880 leader_election.cc:341] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:50.578465 7616 consensus_queue.cc:1035] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:50.579138 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d11cd50b562408f9d3bc24a974bdba8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 08:12:50.580659 4903 leader_election.cc:341] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:50.580387 2072 raft_consensus.cc:2932] T 9d11cd50b562408f9d3bc24a974bdba8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:50.582585 1664 catalog_manager.cc:5484] T 9d11cd50b562408f9d3bc24a974bdba8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:50.589526 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6994370a532472893cc934337f3cb6e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:50.589835 4568 raft_consensus.cc:3037] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.594246 4967 raft_consensus.cc:2932] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:50.598903 4568 raft_consensus.cc:2454] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:50.599608 4900 leader_election.cc:302] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:50.599922 7585 raft_consensus.cc:2781] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:50.600082 7585 raft_consensus.cc:686] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:50.600415 7585 consensus_queue.cc:237] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.602644 2184 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d11cd50b562408f9d3bc24a974bdba8 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:50.603008 7590 tablet_replica.cc:330] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:50.603138 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6994370a532472893cc934337f3cb6e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:50.603242 7590 raft_consensus.cc:2229] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:50.603513 7590 raft_consensus.cc:2258] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:12:50.603739 4903 leader_election.cc:341] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:50.610915 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11dc200302b4d64bcdceb32e5171048" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:50.611431 4966 raft_consensus.cc:2454] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:50.612113 1878 leader_election.cc:302] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters: f24648c92c384f1ba349eee801f985e2
I20240503 08:12:50.612779 7618 raft_consensus.cc:2781] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:50.612877 7618 raft_consensus.cc:482] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:50.612952 7618 raft_consensus.cc:3037] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.616786 7618 raft_consensus.cc:504] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.617493 7590 ts_tablet_manager.cc:1875] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:50.617982 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11dc200302b4d64bcdceb32e5171048" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:50.618233 4966 raft_consensus.cc:3037] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.622638 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11dc200302b4d64bcdceb32e5171048" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
W20240503 08:12:50.623657 1880 leader_election.cc:341] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:50.623965 7618 leader_election.cc:288] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:50.624233 7590 ts_tablet_manager.cc:1888] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:12:50.624421 7590 log.cc:1196] T 9d11cd50b562408f9d3bc24a974bdba8 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/9d11cd50b562408f9d3bc24a974bdba8
I20240503 08:12:50.625398 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 9d11cd50b562408f9d3bc24a974bdba8 (table rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]) successfully deleted
I20240503 08:12:50.626816 4966 raft_consensus.cc:2454] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:50.627354 1878 leader_election.cc:302] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters: f24648c92c384f1ba349eee801f985e2
I20240503 08:12:50.627835 7618 raft_consensus.cc:2781] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:50.628356 7618 raft_consensus.cc:686] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:50.628690 7618 consensus_queue.cc:237] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.629984 7627 tablet_bootstrap.cc:492] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:50.630247 7640 tablet_bootstrap.cc:492] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:50.630322 7627 ts_tablet_manager.cc:1367] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.205s user 0.010s sys 0.045s
I20240503 08:12:50.630554 7640 ts_tablet_manager.cc:1367] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.202s user 0.014s sys 0.038s
I20240503 08:12:50.631570 7627 raft_consensus.cc:348] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.631877 7627 raft_consensus.cc:374] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.632001 7627 raft_consensus.cc:729] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.632086 1672 catalog_manager.cc:5484] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:50.632295 7627 consensus_queue.cc:260] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.631623 7640 raft_consensus.cc:348] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:50.632773 7640 raft_consensus.cc:374] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.632882 7640 raft_consensus.cc:729] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.633014 7627 ts_tablet_manager.cc:1398] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:50.633141 7640 consensus_queue.cc:260] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:50.634033 7640 ts_tablet_manager.cc:1398] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:50.632681 1672 catalog_manager.cc:5484] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:50.641036 7627 tablet_bootstrap.cc:492] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:50.643352 7627 tablet_bootstrap.cc:654] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.656899 7627 tablet_bootstrap.cc:492] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:50.657338 7627 ts_tablet_manager.cc:1367] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.010s
I20240503 08:12:50.660352 7366 raft_consensus.cc:482] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.660523 7366 raft_consensus.cc:504] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:50.660861 7627 raft_consensus.cc:348] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.661078 7627 raft_consensus.cc:374] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.661149 7627 raft_consensus.cc:729] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.661319 7366 leader_election.cc:288] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:50.661420 7627 consensus_queue.cc:260] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.662122 7627 ts_tablet_manager.cc:1398] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:50.662421 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0bffb0aa304bd184b8ad26bc94fd18" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:50.662709 7627 tablet_bootstrap.cc:492] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:50.662729 4701 raft_consensus.cc:2379] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b8f581390da449ccb9e299011296405e in current term 1: Already voted for candidate 00b61625d8a7460892c118a2570afda5 in this term.
I20240503 08:12:50.666914 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0bffb0aa304bd184b8ad26bc94fd18" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:50.668025 1747 leader_election.cc:302] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f581390da449ccb9e299011296405e; no voters: 00b61625d8a7460892c118a2570afda5, 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:50.669025 7366 raft_consensus.cc:3037] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.669915 7627 tablet_bootstrap.cc:654] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:50.673458 7366 raft_consensus.cc:2726] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:50.681898 7627 tablet_bootstrap.cc:492] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:50.682193 7627 ts_tablet_manager.cc:1367] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.004s
I20240503 08:12:50.687520 7627 raft_consensus.cc:348] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.687901 7627 raft_consensus.cc:374] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:50.688062 7627 raft_consensus.cc:729] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:50.688388 7627 consensus_queue.cc:260] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.688880 7606 raft_consensus.cc:482] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.689030 7606 raft_consensus.cc:504] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.689177 7627 ts_tablet_manager.cc:1398] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:50.689757 7606 leader_election.cc:288] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:50.690910 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11dc200302b4d64bcdceb32e5171048" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:50.691008 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11dc200302b4d64bcdceb32e5171048" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:50.691391 4967 raft_consensus.cc:2379] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f24648c92c384f1ba349eee801f985e2 in current term 1: Already voted for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in this term.
I20240503 08:12:50.692080 4768 leader_election.cc:302] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f24648c92c384f1ba349eee801f985e2; no voters: 00b61625d8a7460892c118a2570afda5, 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:12:50.692602 7606 raft_consensus.cc:3037] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.696282 7606 raft_consensus.cc:2726] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:50.703289 7607 raft_consensus.cc:482] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.703502 7607 raft_consensus.cc:504] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.704313 7607 leader_election.cc:288] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:50.705355 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ad826dca7c4343afcae34bf1a40da8" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:50.705643 4568 raft_consensus.cc:2454] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:50.706432 4767 leader_election.cc:302] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:50.706494 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ad826dca7c4343afcae34bf1a40da8" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:50.706744 4703 raft_consensus.cc:2454] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:50.707465 7607 raft_consensus.cc:2781] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:50.707595 7607 raft_consensus.cc:482] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:50.707697 7607 raft_consensus.cc:3037] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.709962 7607 raft_consensus.cc:504] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.710937 7607 leader_election.cc:288] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:50.711313 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ad826dca7c4343afcae34bf1a40da8" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:50.711583 4703 raft_consensus.cc:3037] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.709338 1668 catalog_manager.cc:5484] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:50.715564 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ad826dca7c4343afcae34bf1a40da8" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:50.715847 4568 raft_consensus.cc:3037] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.717936 4703 raft_consensus.cc:2454] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:50.718603 4568 raft_consensus.cc:2454] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:50.719098 4767 leader_election.cc:302] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:50.719503 7607 raft_consensus.cc:2781] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:50.719669 7607 raft_consensus.cc:686] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:50.720000 7607 consensus_queue.cc:237] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.728229 1671 catalog_manager.cc:5484] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:50.817127 7606 raft_consensus.cc:482] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:50.817356 7606 raft_consensus.cc:504] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:50.818032 7606 leader_election.cc:288] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:50.818885 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6994370a532472893cc934337f3cb6e" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:50.819169 4568 raft_consensus.cc:2379] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f24648c92c384f1ba349eee801f985e2 in current term 1: Already voted for candidate 00b61625d8a7460892c118a2570afda5 in this term.
I20240503 08:12:50.819373 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6994370a532472893cc934337f3cb6e" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:50.820092 4768 leader_election.cc:302] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f24648c92c384f1ba349eee801f985e2; no voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:50.820453 7606 raft_consensus.cc:3037] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:50.823359 7606 raft_consensus.cc:2726] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:50.848445 1671 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:50.854784 4967 raft_consensus.cc:1261] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:50.855470 4834 raft_consensus.cc:1261] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:50.855459 7618 consensus_queue.cc:1035] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.856906 7618 consensus_queue.cc:1035] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:50.875299 7631 mvcc.cc:205] Tried to move back new op lower bound from 7023509384614985728 to 7023509383746252800. Current Snapshot: MvccSnapshot[applied={T|T < 7023509384614985728}]
I20240503 08:12:50.875800 7708 tablet.cc:1722] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:50.878176 7709 tablet.cc:1722] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:50.878793 7709 tablet.cc:1620] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:50.879125 1671 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet c11dc200302b4d64bcdceb32e5171048 (table rebalancer_dynamic_table [id=8a49b8cedbec4f2a8efa8bb7c5a902cd]). Expected version 1 got 0
I20240503 08:12:50.879487 7708 tablet.cc:1620] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: MemRowSet was empty: no flush needed.
I20240503 08:12:50.886194 1668 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet c11dc200302b4d64bcdceb32e5171048 (table rebalancer_dynamic_table [id=8a49b8cedbec4f2a8efa8bb7c5a902cd]). Expected version 1 got 0
I20240503 08:12:50.893640 1664 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=8a49b8cedbec4f2a8efa8bb7c5a902cd] alter complete (version 1)
I20240503 08:12:50.905396 7712 tablet.cc:1722] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:50.907114 7712 tablet.cc:1620] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:50.908437 1665 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet c11dc200302b4d64bcdceb32e5171048 (table rebalancer_dynamic_table [id=8a49b8cedbec4f2a8efa8bb7c5a902cd]). Expected version 1 got 0
I20240503 08:12:50.996152 7585 consensus_queue.cc:1035] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.001821 7184 consensus_queue.cc:1035] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.016707 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:51.034446 7708 tablet.cc:1722] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:51.035043 7708 tablet.cc:1620] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: MemRowSet was empty: no flush needed.
I20240503 08:12:51.039526 7709 tablet.cc:1722] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:51.040112 7709 tablet.cc:1620] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:51.040665 7712 tablet.cc:1722] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:51.041174 7712 tablet.cc:1620] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:51.062131 1672 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=8a49b8cedbec4f2a8efa8bb7c5a902cd] alter complete (version 2)
I20240503 08:12:51.112175 7607 consensus_queue.cc:1035] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.136490 7511 consensus_queue.cc:1035] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.136965 7607 consensus_queue.cc:1035] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:51.169823 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:51.188681 7708 tablet.cc:1722] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:51.192513 7709 tablet.cc:1722] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:51.197307 7585 consensus_queue.cc:1035] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:51.204833 1748 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.194:41975, user_credentials={real_user=slave}} blocked reactor thread for 47023us
I20240503 08:12:51.212267 7712 tablet.cc:1722] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:51.224963 1667 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=8a49b8cedbec4f2a8efa8bb7c5a902cd] alter complete (version 3)
I20240503 08:12:51.314989 4683 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:12:51.316579 1923 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:12:51.328568 4945 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:12:51.328698 2184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:12:51.332509 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:51.338133 1667 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:51.338362 1667 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:51.342902 1667 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet c11dc200302b4d64bcdceb32e5171048
I20240503 08:12:51.345981 4814 tablet_service.cc:1514] Processing DeleteTablet for tablet c11dc200302b4d64bcdceb32e5171048 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:51 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:51.346618 4945 tablet_service.cc:1514] Processing DeleteTablet for tablet c11dc200302b4d64bcdceb32e5171048 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:51 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:51.349165 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet c11dc200302b4d64bcdceb32e5171048 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:51 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:51.350018 2184 tablet_service.cc:1467] Processing CreateTablet for tablet 5d7781b8af904afbb8773a9e2f860388 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1f92ea0129af44199b7cc48b5849c7ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.350033 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 5d7781b8af904afbb8773a9e2f860388 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1f92ea0129af44199b7cc48b5849c7ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.350492 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d7781b8af904afbb8773a9e2f860388. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.350504 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d7781b8af904afbb8773a9e2f860388. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.352005 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 05274a8ee1ac4cada1f7788f43d7f1f3 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1f92ea0129af44199b7cc48b5849c7ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.352459 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05274a8ee1ac4cada1f7788f43d7f1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.354314 4946 tablet_service.cc:1467] Processing CreateTablet for tablet 5d7781b8af904afbb8773a9e2f860388 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1f92ea0129af44199b7cc48b5849c7ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.354918 4946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d7781b8af904afbb8773a9e2f860388. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.358616 4947 tablet_service.cc:1467] Processing CreateTablet for tablet 05274a8ee1ac4cada1f7788f43d7f1f3 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1f92ea0129af44199b7cc48b5849c7ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.359076 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05274a8ee1ac4cada1f7788f43d7f1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.362411 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba03a430b4b498f9ba66d733d68b13c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1f92ea0129af44199b7cc48b5849c7ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.362937 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba03a430b4b498f9ba66d733d68b13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.366447 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 05274a8ee1ac4cada1f7788f43d7f1f3 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1f92ea0129af44199b7cc48b5849c7ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.366966 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05274a8ee1ac4cada1f7788f43d7f1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.378352 4944 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba03a430b4b498f9ba66d733d68b13c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1f92ea0129af44199b7cc48b5849c7ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.378911 4944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba03a430b4b498f9ba66d733d68b13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.398073 7765 tablet_bootstrap.cc:492] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:51.399518 7762 tablet_replica.cc:330] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:51.399832 7762 raft_consensus.cc:2229] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:51.402961 7762 raft_consensus.cc:2258] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:51.404044 7766 tablet_bootstrap.cc:492] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:51.410181 7766 tablet_bootstrap.cc:654] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.410544 7765 tablet_bootstrap.cc:654] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.410753 7762 ts_tablet_manager.cc:1875] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:51.416661 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba03a430b4b498f9ba66d733d68b13c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=1f92ea0129af44199b7cc48b5849c7ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:51.417224 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba03a430b4b498f9ba66d733d68b13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.422104 2052 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:12:51.438999 7761 tablet_replica.cc:330] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:51.439581 7761 raft_consensus.cc:2229] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:51.439915 7761 raft_consensus.cc:2258] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:51.441327 7761 ts_tablet_manager.cc:1875] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:12:51.442983 4504 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.199:39455, user_credentials={real_user=slave}} blocked reactor thread for 104083us
I20240503 08:12:51.446352 1667 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:51.449795 7760 tablet_replica.cc:330] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:51.450124 7760 raft_consensus.cc:2229] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:51.450448 7760 raft_consensus.cc:2258] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:51.454722 7762 ts_tablet_manager.cc:1888] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:51.455024 7762 log.cc:1196] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/c11dc200302b4d64bcdceb32e5171048
I20240503 08:12:51.455731 7762 ts_tablet_manager.cc:1909] T c11dc200302b4d64bcdceb32e5171048 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:51.456940 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet c11dc200302b4d64bcdceb32e5171048 (table rebalancer_dynamic_table_new_name [id=8a49b8cedbec4f2a8efa8bb7c5a902cd]) successfully deleted
I20240503 08:12:51.457705 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 003c33028f594c788b84508f85a180c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8680c4f223ae4b388eaaa493392d7380]), partition=
I20240503 08:12:51.457820 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 003c33028f594c788b84508f85a180c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8680c4f223ae4b388eaaa493392d7380]), partition=
I20240503 08:12:51.458016 7768 tablet_bootstrap.cc:492] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:51.458050 7760 ts_tablet_manager.cc:1875] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:51.458099 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003c33028f594c788b84508f85a180c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.458186 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003c33028f594c788b84508f85a180c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.461722 7768 tablet_bootstrap.cc:654] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.464054 7771 tablet_bootstrap.cc:492] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:51.467569 7765 tablet_bootstrap.cc:492] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:51.467846 7765 ts_tablet_manager.cc:1367] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.000s
I20240503 08:12:51.468633 7765 raft_consensus.cc:348] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.469004 7765 raft_consensus.cc:374] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.469128 7765 raft_consensus.cc:729] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.469259 4547 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:12:51.469486 7765 consensus_queue.cc:260] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.471421 7765 ts_tablet_manager.cc:1398] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:51.471832 7765 tablet_bootstrap.cc:492] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:51.473791 7765 tablet_bootstrap.cc:654] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.476869 7766 tablet_bootstrap.cc:492] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:51.477119 7766 ts_tablet_manager.cc:1367] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.073s user 0.010s sys 0.000s
I20240503 08:12:51.478415 7766 raft_consensus.cc:348] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.478761 7766 raft_consensus.cc:374] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.478904 7766 raft_consensus.cc:729] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.479185 7766 consensus_queue.cc:260] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.480763 7766 ts_tablet_manager.cc:1398] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:51.489734 7771 tablet_bootstrap.cc:654] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.489835 7765 tablet_bootstrap.cc:492] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:51.490278 7765 ts_tablet_manager.cc:1367] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20240503 08:12:51.493924 7765 raft_consensus.cc:348] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:51.494994 7765 raft_consensus.cc:374] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.495249 7765 raft_consensus.cc:729] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.495834 7765 consensus_queue.cc:260] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:51.498071 7765 ts_tablet_manager.cc:1398] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:12:51.500967 7765 tablet_bootstrap.cc:492] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:51.505203 7765 tablet_bootstrap.cc:654] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.509188 7760 ts_tablet_manager.cc:1888] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:51.515440 7760 log.cc:1196] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/c11dc200302b4d64bcdceb32e5171048
I20240503 08:12:51.517513 7760 ts_tablet_manager.cc:1909] T c11dc200302b4d64bcdceb32e5171048 P f24648c92c384f1ba349eee801f985e2: Deleting consensus metadata
I20240503 08:12:51.521916 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet c11dc200302b4d64bcdceb32e5171048 (table rebalancer_dynamic_table_new_name [id=8a49b8cedbec4f2a8efa8bb7c5a902cd]) successfully deleted
I20240503 08:12:51.524102 7767 tablet_bootstrap.cc:492] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:51.530745 7767 tablet_bootstrap.cc:654] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.533036 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 003c33028f594c788b84508f85a180c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8680c4f223ae4b388eaaa493392d7380]), partition=
I20240503 08:12:51.533699 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003c33028f594c788b84508f85a180c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:51.541237 7771 tablet_bootstrap.cc:492] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:51.541507 7771 ts_tablet_manager.cc:1367] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.005s
I20240503 08:12:51.541801 7768 tablet_bootstrap.cc:492] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:51.542111 7768 ts_tablet_manager.cc:1367] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.084s user 0.007s sys 0.000s
I20240503 08:12:51.542541 7771 raft_consensus.cc:348] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.542830 7771 raft_consensus.cc:374] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.542968 7771 raft_consensus.cc:729] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.544713 7768 raft_consensus.cc:348] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.545003 7768 raft_consensus.cc:374] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.545147 7768 raft_consensus.cc:729] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.545058 7771 consensus_queue.cc:260] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.545435 7768 consensus_queue.cc:260] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.546062 7768 ts_tablet_manager.cc:1398] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:51.547816 7771 ts_tablet_manager.cc:1398] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:51.548156 7771 tablet_bootstrap.cc:492] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:51.549985 7771 tablet_bootstrap.cc:654] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.563721 1793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:12:51.565213 4812 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:12:51.568691 7771 tablet_bootstrap.cc:492] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:51.568892 7771 ts_tablet_manager.cc:1367] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20240503 08:12:51.571663 7771 raft_consensus.cc:348] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:51.574527 7771 raft_consensus.cc:374] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.574627 7771 raft_consensus.cc:729] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.574990 7771 consensus_queue.cc:260] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:51.582093 7771 ts_tablet_manager.cc:1398] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240503 08:12:51.586158 7771 tablet_bootstrap.cc:492] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:51.591341 7767 tablet_bootstrap.cc:492] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:51.591635 7767 ts_tablet_manager.cc:1367] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.068s user 0.007s sys 0.000s
I20240503 08:12:51.592577 7767 raft_consensus.cc:348] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:51.592979 7767 raft_consensus.cc:374] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.593118 7767 raft_consensus.cc:729] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.593430 7767 consensus_queue.cc:260] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:51.594645 7771 tablet_bootstrap.cc:654] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.599659 7767 ts_tablet_manager.cc:1398] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:12:51.602350 7767 tablet_bootstrap.cc:492] T 5d7781b8af904afbb8773a9e2f860388 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:51.608672 7765 tablet_bootstrap.cc:492] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:51.609845 7771 tablet_bootstrap.cc:492] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:51.610050 7771 ts_tablet_manager.cc:1367] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.003s
I20240503 08:12:51.610183 7765 ts_tablet_manager.cc:1367] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.003s
I20240503 08:12:51.611223 7771 raft_consensus.cc:348] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.611471 7771 raft_consensus.cc:374] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.611570 7771 raft_consensus.cc:729] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.611977 7771 consensus_queue.cc:260] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.612517 7771 ts_tablet_manager.cc:1398] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:51.614436 7765 raft_consensus.cc:348] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.623422 7765 raft_consensus.cc:374] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.611418 7767 tablet_bootstrap.cc:654] T 5d7781b8af904afbb8773a9e2f860388 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.624761 7765 raft_consensus.cc:729] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.625061 7765 consensus_queue.cc:260] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.625658 7765 ts_tablet_manager.cc:1398] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240503 08:12:51.606385 7761 ts_tablet_manager.cc:1888] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:51.626324 7761 log.cc:1196] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/c11dc200302b4d64bcdceb32e5171048
I20240503 08:12:51.627183 7761 ts_tablet_manager.cc:1909] T c11dc200302b4d64bcdceb32e5171048 P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:51.628423 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet c11dc200302b4d64bcdceb32e5171048 (table rebalancer_dynamic_table_new_name [id=8a49b8cedbec4f2a8efa8bb7c5a902cd]) successfully deleted
I20240503 08:12:51.631549 7502 raft_consensus.cc:482] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:51.631726 7502 raft_consensus.cc:504] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.632634 7502 leader_election.cc:288] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:51.633261 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d7781b8af904afbb8773a9e2f860388" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:51.634009 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d7781b8af904afbb8773a9e2f860388" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:51.634227 2202 raft_consensus.cc:2454] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:51.634975 4636 leader_election.cc:302] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters:
W20240503 08:12:51.635362 4636 leader_election.cc:341] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:51.636538 7767 tablet_bootstrap.cc:492] T 5d7781b8af904afbb8773a9e2f860388 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:51.636744 7767 ts_tablet_manager.cc:1367] T 5d7781b8af904afbb8773a9e2f860388 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240503 08:12:51.637651 7767 raft_consensus.cc:348] T 5d7781b8af904afbb8773a9e2f860388 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.637899 7767 raft_consensus.cc:374] T 5d7781b8af904afbb8773a9e2f860388 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.637981 7767 raft_consensus.cc:729] T 5d7781b8af904afbb8773a9e2f860388 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.638234 7767 consensus_queue.cc:260] T 5d7781b8af904afbb8773a9e2f860388 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.638845 7767 ts_tablet_manager.cc:1398] T 5d7781b8af904afbb8773a9e2f860388 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:51.639914 7502 raft_consensus.cc:2781] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:51.640036 7502 raft_consensus.cc:482] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:51.640127 7502 raft_consensus.cc:3037] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.641415 7767 tablet_bootstrap.cc:492] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:51.643429 7767 tablet_bootstrap.cc:654] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.652885 7502 raft_consensus.cc:504] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.653692 7502 leader_election.cc:288] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:51.654458 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d7781b8af904afbb8773a9e2f860388" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:51.654707 4966 raft_consensus.cc:3037] T 5d7781b8af904afbb8773a9e2f860388 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.654832 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d7781b8af904afbb8773a9e2f860388" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:51.655026 2202 raft_consensus.cc:3037] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.658010 2202 raft_consensus.cc:2454] T 5d7781b8af904afbb8773a9e2f860388 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:51.658490 4636 leader_election.cc:302] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:51.658859 7502 raft_consensus.cc:2781] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:51.659032 7502 raft_consensus.cc:686] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:51.659405 7502 consensus_queue.cc:237] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.662290 4966 raft_consensus.cc:2454] T 5d7781b8af904afbb8773a9e2f860388 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:51.665134 7767 tablet_bootstrap.cc:492] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:51.667485 7767 ts_tablet_manager.cc:1367] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.007s
I20240503 08:12:51.668329 7767 raft_consensus.cc:348] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.671030 7706 raft_consensus.cc:482] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:51.671296 7767 raft_consensus.cc:374] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.671249 7706 raft_consensus.cc:504] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:51.671494 7767 raft_consensus.cc:729] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.673347 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba03a430b4b498f9ba66d733d68b13c" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:51.674708 4966 raft_consensus.cc:2454] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:51.675351 7767 consensus_queue.cc:260] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.678787 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba03a430b4b498f9ba66d733d68b13c" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:51.679075 4699 raft_consensus.cc:2454] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 0.
I20240503 08:12:51.679878 2138 leader_election.cc:302] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 184c543cb05d49cc98ba1c56617be7a5; no voters:
I20240503 08:12:51.680168 7767 ts_tablet_manager.cc:1398] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240503 08:12:51.680239 7129 raft_consensus.cc:2781] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:51.680317 7129 raft_consensus.cc:482] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:51.680379 7129 raft_consensus.cc:3037] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.680460 7767 tablet_bootstrap.cc:492] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:51.686148 7129 raft_consensus.cc:504] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:51.688005 7767 tablet_bootstrap.cc:654] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:51.688524 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba03a430b4b498f9ba66d733d68b13c" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:51.688781 4699 raft_consensus.cc:3037] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.692692 4699 raft_consensus.cc:2454] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:51.693333 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba03a430b4b498f9ba66d733d68b13c" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:51.693601 4966 raft_consensus.cc:3037] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.697341 2140 leader_election.cc:302] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:51.700881 7129 leader_election.cc:288] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:51.702451 7129 raft_consensus.cc:2781] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:51.704739 7129 raft_consensus.cc:686] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Running, Role: LEADER
I20240503 08:12:51.705384 7129 consensus_queue.cc:237] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:51.706192 4966 raft_consensus.cc:2454] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184c543cb05d49cc98ba1c56617be7a5 in term 1.
I20240503 08:12:51.701133 7706 leader_election.cc:288] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:51.709710 7767 tablet_bootstrap.cc:492] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:51.709936 7767 ts_tablet_manager.cc:1367] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.006s
I20240503 08:12:51.710706 7767 raft_consensus.cc:348] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.711390 7767 raft_consensus.cc:374] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:51.711542 7767 raft_consensus.cc:729] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:51.712178 7767 consensus_queue.cc:260] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.714627 7767 ts_tablet_manager.cc:1398] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 08:12:51.723558 1667 catalog_manager.cc:5484] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:51.766434 1667 catalog_manager.cc:5484] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:51.798240 7606 raft_consensus.cc:482] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:51.798425 7606 raft_consensus.cc:504] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.799191 7606 leader_election.cc:288] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:51.801587 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05274a8ee1ac4cada1f7788f43d7f1f3" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:51.801808 1944 raft_consensus.cc:2454] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:51.803488 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05274a8ee1ac4cada1f7788f43d7f1f3" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:51.803740 4966 raft_consensus.cc:2454] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:51.804246 4768 leader_election.cc:302] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:51.805572 7606 raft_consensus.cc:2781] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:51.805734 7606 raft_consensus.cc:482] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:51.805830 7606 raft_consensus.cc:3037] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.808398 7606 raft_consensus.cc:504] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.809211 7606 leader_election.cc:288] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:51.809538 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05274a8ee1ac4cada1f7788f43d7f1f3" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:51.809670 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05274a8ee1ac4cada1f7788f43d7f1f3" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:51.809803 1944 raft_consensus.cc:3037] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.809907 4966 raft_consensus.cc:3037] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.812716 4966 raft_consensus.cc:2454] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:51.813467 4768 leader_election.cc:302] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:51.813822 7606 raft_consensus.cc:2781] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:51.813998 7606 raft_consensus.cc:686] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:51.814908 7606 consensus_queue.cc:237] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.820647 1944 raft_consensus.cc:2454] T 05274a8ee1ac4cada1f7788f43d7f1f3 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:51.824940 1667 catalog_manager.cc:5484] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:51.861181 4566 consensus_queue.cc:237] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:51.864151 4699 raft_consensus.cc:1261] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:51.864374 1944 raft_consensus.cc:1261] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:51.864830 7354 consensus_queue.cc:1035] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:12:51.865208 4502 consensus_peers.cc:485] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b48714749469474886f9e1d7e7635915. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:51.865250 7763 consensus_queue.cc:1035] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:51.868682 7759 raft_consensus.cc:2932] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:51.869498 4699 raft_consensus.cc:2932] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:51.871939 7502 raft_consensus.cc:482] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:51.872134 7502 raft_consensus.cc:504] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.872743 7502 leader_election.cc:288] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:51.873263 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003c33028f594c788b84508f85a180c9" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:51.873495 4966 raft_consensus.cc:2454] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:51.873469 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003c33028f594c788b84508f85a180c9" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:51.873674 2204 raft_consensus.cc:2454] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:51.873935 4636 leader_election.cc:302] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:51.874208 7179 raft_consensus.cc:2781] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:51.874326 7179 raft_consensus.cc:482] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:51.874449 7179 raft_consensus.cc:3037] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.875533 1944 raft_consensus.cc:2932] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:51.877199 7179 raft_consensus.cc:504] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.877048 1667 catalog_manager.cc:5484] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:51.877838 7179 leader_election.cc:288] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:51.878567 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003c33028f594c788b84508f85a180c9" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:51.878760 2204 raft_consensus.cc:3037] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.878751 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003c33028f594c788b84508f85a180c9" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:51.878990 4966 raft_consensus.cc:3037] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:51.882012 4966 raft_consensus.cc:2454] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:51.882553 4636 leader_election.cc:302] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:51.882638 2204 raft_consensus.cc:2454] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:51.882995 7179 raft_consensus.cc:2781] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:51.883292 7179 raft_consensus.cc:686] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:51.883832 7179 consensus_queue.cc:237] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:51.892055 1665 catalog_manager.cc:5484] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:51.902734 4699 consensus_queue.cc:237] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:51.905858 4566 raft_consensus.cc:1261] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:51.905977 4835 raft_consensus.cc:1261] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:51.906692 7179 consensus_queue.cc:1035] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:12:51.907239 4637 consensus_peers.cc:485] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53ce7a6567274050a2b377a9c650ccc4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:51.907655 7502 consensus_queue.cc:1035] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:51.910130 7805 raft_consensus.cc:2932] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:51.910828 4566 raft_consensus.cc:2932] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:51.911767 4835 raft_consensus.cc:2932] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:51.917716 1665 catalog_manager.cc:5484] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:51.942457 4699 consensus_queue.cc:237] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:51.945242 4966 raft_consensus.cc:1261] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:51.945344 2072 raft_consensus.cc:1261] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:12:51.945508 4638 consensus_peers.cc:485] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40476d1762154f6d9ce4855872492573. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:51.945842 7805 consensus_queue.cc:1035] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:51.946159 7502 consensus_queue.cc:1035] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:51.949223 7502 raft_consensus.cc:2932] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:51.950042 4966 raft_consensus.cc:2932] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:51.956566 1665 catalog_manager.cc:5484] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index -1 to 2, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:51.968132 2072 raft_consensus.cc:2932] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:51.998900 4835 consensus_queue.cc:237] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:52.001703 2204 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 08:12:52.001962 4770 consensus_peers.cc:485] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54fe406b5f324d629243f7f939535620. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:52.002759 2072 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:12:52.002810 7606 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:52.003618 7606 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:12:52.008558 7742 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:52.009183 2073 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:52.016762 1665 catalog_manager.cc:5484] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 4 to 5, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:52.024217 2204 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:52.062331 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:52.068238 1810 consensus_queue.cc:237] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:52.074594 1944 raft_consensus.cc:1261] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.075014 4966 raft_consensus.cc:1261] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.075862 7734 consensus_queue.cc:1035] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:52.076457 7734 consensus_queue.cc:1035] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:12:52.077440 1747 consensus_peers.cc:485] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d1e2530d5f2431baf0b2dd41081bb9d. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:52.088825 4967 raft_consensus.cc:1261] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.089705 7805 consensus_queue.cc:1035] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.095860 7737 raft_consensus.cc:2932] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:52.100780 1941 raft_consensus.cc:2932] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:52.110853 7785 consensus_queue.cc:1035] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.111101 1665 catalog_manager.cc:5484] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:52.125233 7129 consensus_queue.cc:1035] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.143555 2202 raft_consensus.cc:1261] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.152410 7179 consensus_queue.cc:1035] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.157589 7805 consensus_queue.cc:1035] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.167811 7805 consensus_queue.cc:1035] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.210500 4966 raft_consensus.cc:2932] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
W20240503 08:12:52.221112 2008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.196:42183, user_credentials={real_user=slave}} blocked reactor thread for 89044.8us
I20240503 08:12:52.237790 1665 catalog_manager.cc:5576] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183) does not have the latest schema for tablet 003c33028f594c788b84508f85a180c9 (table rebalancer_dynamic_table [id=8680c4f223ae4b388eaaa493392d7380]). Expected version 1 got 0
I20240503 08:12:52.242727 1667 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 003c33028f594c788b84508f85a180c9 (table rebalancer_dynamic_table [id=8680c4f223ae4b388eaaa493392d7380]). Expected version 1 got 0
I20240503 08:12:52.248056 7606 consensus_queue.cc:1035] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.262742 7854 tablet.cc:1722] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:52.263455 7854 tablet.cc:1620] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:52.265666 7851 tablet.cc:1722] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:52.266898 7851 tablet.cc:1620] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:52.267726 1665 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 003c33028f594c788b84508f85a180c9 (table rebalancer_dynamic_table [id=8680c4f223ae4b388eaaa493392d7380]). Expected version 1 got 0
I20240503 08:12:52.267736 7853 tablet.cc:1722] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:52.268347 7853 tablet.cc:1620] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: MemRowSet was empty: no flush needed.
I20240503 08:12:52.274994 1667 catalog_manager.cc:5576] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183) does not have the latest schema for tablet 003c33028f594c788b84508f85a180c9 (table rebalancer_dynamic_table [id=8680c4f223ae4b388eaaa493392d7380]). Expected version 1 got 0
I20240503 08:12:52.282897 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=8680c4f223ae4b388eaaa493392d7380] alter complete (version 1)
I20240503 08:12:52.298134 4965 consensus_queue.cc:237] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
W20240503 08:12:52.301836 4902 consensus_peers.cc:485] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34b7fd8dc7434b8ea14b9deb0023b4cb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:52.304780 4566 raft_consensus.cc:1261] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.305500 7585 consensus_queue.cc:1035] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:52.307818 7606 consensus_queue.cc:1035] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:52.308193 1943 raft_consensus.cc:1261] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.309619 7585 consensus_queue.cc:1035] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:52.321323 7585 raft_consensus.cc:2932] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:52.324342 4568 raft_consensus.cc:2932] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:52.334857 1943 raft_consensus.cc:2932] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:52.336064 1667 catalog_manager.cc:5484] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:52.358721 7868 ts_tablet_manager.cc:902] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:52.360790 7868 tablet_copy_client.cc:308] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.198:46559
I20240503 08:12:52.361845 4723 tablet_copy_service.cc:140] P 31f41542452b4b1eb994ff4223d039fc: Received BeginTabletCopySession request for tablet 53ce7a6567274050a2b377a9c650ccc4 from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:41709)
I20240503 08:12:52.362130 4723 tablet_copy_service.cc:161] P 31f41542452b4b1eb994ff4223d039fc: Beginning new tablet copy session on tablet 53ce7a6567274050a2b377a9c650ccc4 from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:41709: session id = b8f581390da449ccb9e299011296405e-53ce7a6567274050a2b377a9c650ccc4
I20240503 08:12:52.365833 4723 tablet_copy_source_session.cc:205] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:52.367568 7873 ts_tablet_manager.cc:902] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:52.367935 7868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ce7a6567274050a2b377a9c650ccc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.367815 4566 consensus_queue.cc:237] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:52.371934 4835 raft_consensus.cc:1261] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.372270 4965 raft_consensus.cc:1261] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:12:52.372534 4502 consensus_peers.cc:485] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84831077defe434c9f29eecfae7b44c8. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:52.372891 7876 consensus_queue.cc:1035] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:52.373302 7764 consensus_queue.cc:1035] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:52.375681 7868 tablet_copy_client.cc:783] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:52.376067 7868 tablet_copy_client.cc:647] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:52.379977 7868 tablet_copy_client.cc:523] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:52.380344 1667 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:52.385063 7868 tablet_bootstrap.cc:492] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:52.395046 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 241f60a993f94046b870eae6ba173cc1 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b707fc1303e446cbc4639468deac7b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.395578 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241f60a993f94046b870eae6ba173cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.395985 1924 tablet_service.cc:1467] Processing CreateTablet for tablet a7e0415e028b4303bcf4ab64f4ec6b54 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b707fc1303e446cbc4639468deac7b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.396118 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 241f60a993f94046b870eae6ba173cc1 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b707fc1303e446cbc4639468deac7b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.396387 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e0415e028b4303bcf4ab64f4ec6b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.396610 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241f60a993f94046b870eae6ba173cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.400126 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 241f60a993f94046b870eae6ba173cc1 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b707fc1303e446cbc4639468deac7b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.400173 2052 tablet_service.cc:1467] Processing CreateTablet for tablet a7e0415e028b4303bcf4ab64f4ec6b54 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b707fc1303e446cbc4639468deac7b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.400614 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 241f60a993f94046b870eae6ba173cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.400707 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e0415e028b4303bcf4ab64f4ec6b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.404310 1923 tablet_service.cc:1467] Processing CreateTablet for tablet 3ebf60954ed44e29b5f8047ed511d2c7 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b707fc1303e446cbc4639468deac7b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.404346 4547 tablet_service.cc:1467] Processing CreateTablet for tablet a7e0415e028b4303bcf4ab64f4ec6b54 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b707fc1303e446cbc4639468deac7b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.404786 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ebf60954ed44e29b5f8047ed511d2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.404829 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e0415e028b4303bcf4ab64f4ec6b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.406783 4944 tablet_service.cc:1467] Processing CreateTablet for tablet 3ebf60954ed44e29b5f8047ed511d2c7 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b707fc1303e446cbc4639468deac7b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.407439 4944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ebf60954ed44e29b5f8047ed511d2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.409935 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 3ebf60954ed44e29b5f8047ed511d2c7 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b707fc1303e446cbc4639468deac7b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:52.410524 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ebf60954ed44e29b5f8047ed511d2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.411396 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:52.415310 7764 raft_consensus.cc:2932] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:52.421653 4835 raft_consensus.cc:2932] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:52.430482 1667 catalog_manager.cc:5484] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:52.433990 7851 tablet.cc:1722] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:52.433995 7854 tablet.cc:1722] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:52.434538 7851 tablet.cc:1620] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:52.434681 7854 tablet.cc:1620] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:52.451869 7853 tablet.cc:1722] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:52.452430 7853 tablet.cc:1620] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: MemRowSet was empty: no flush needed.
I20240503 08:12:52.457966 4966 raft_consensus.cc:2932] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:52.472121 7883 tablet_bootstrap.cc:492] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:52.474053 7883 tablet_bootstrap.cc:654] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.474898 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=8680c4f223ae4b388eaaa493392d7380] alter complete (version 2)
I20240503 08:12:52.476634 7882 tablet_bootstrap.cc:492] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:52.484988 7882 tablet_bootstrap.cc:654] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.487047 7885 tablet_bootstrap.cc:492] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:52.487648 7881 tablet_bootstrap.cc:492] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:52.491917 7881 tablet_bootstrap.cc:654] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.494453 7883 tablet_bootstrap.cc:492] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:52.494720 7885 tablet_bootstrap.cc:654] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.494752 7883 ts_tablet_manager.cc:1367] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20240503 08:12:52.495630 7883 raft_consensus.cc:348] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:52.495995 7883 raft_consensus.cc:374] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.496177 7883 raft_consensus.cc:729] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.496697 7883 consensus_queue.cc:260] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:52.497623 7883 ts_tablet_manager.cc:1398] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:52.498569 7882 tablet_bootstrap.cc:492] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:52.498842 7882 ts_tablet_manager.cc:1367] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240503 08:12:52.500012 7882 raft_consensus.cc:348] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:52.502619 7882 raft_consensus.cc:374] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.502817 7882 raft_consensus.cc:729] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.503237 7882 consensus_queue.cc:260] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:52.504160 7882 ts_tablet_manager.cc:1398] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:52.504635 7883 tablet_bootstrap.cc:492] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:52.525290 7881 tablet_bootstrap.cc:492] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:52.525590 7881 ts_tablet_manager.cc:1367] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240503 08:12:52.526557 7881 raft_consensus.cc:348] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:52.526895 7881 raft_consensus.cc:374] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.527000 7881 raft_consensus.cc:729] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.527289 7881 consensus_queue.cc:260] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:52.529820 7881 ts_tablet_manager.cc:1398] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:52.530207 7881 tablet_bootstrap.cc:492] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:52.533540 7883 tablet_bootstrap.cc:654] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.536571 7885 tablet_bootstrap.cc:492] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:52.536706 7881 tablet_bootstrap.cc:654] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.536854 7885 ts_tablet_manager.cc:1367] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.005s
I20240503 08:12:52.542508 7885 raft_consensus.cc:348] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:52.542912 7885 raft_consensus.cc:374] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.546895 7885 raft_consensus.cc:729] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.547417 7885 consensus_queue.cc:260] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:52.548859 7883 tablet_bootstrap.cc:492] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:52.549122 7883 ts_tablet_manager.cc:1367] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.000s
I20240503 08:12:52.550107 7883 raft_consensus.cc:348] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:52.550465 7883 raft_consensus.cc:374] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.552230 7883 raft_consensus.cc:729] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.552479 7885 ts_tablet_manager.cc:1398] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 08:12:52.553783 7883 consensus_queue.cc:260] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:52.556460 7883 ts_tablet_manager.cc:1398] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:52.562408 7881 tablet_bootstrap.cc:492] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:52.562729 7881 ts_tablet_manager.cc:1367] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.004s
I20240503 08:12:52.563617 7881 raft_consensus.cc:348] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:52.563926 7881 raft_consensus.cc:374] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.564049 7881 raft_consensus.cc:729] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.564366 7881 consensus_queue.cc:260] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:52.566602 7881 ts_tablet_manager.cc:1398] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:52.567714 7742 raft_consensus.cc:482] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:52.568663 7742 raft_consensus.cc:504] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:52.569371 7742 leader_election.cc:288] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:52.570206 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241f60a993f94046b870eae6ba173cc1" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:52.570436 4965 raft_consensus.cc:2454] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:52.570966 4768 leader_election.cc:302] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:52.571653 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241f60a993f94046b870eae6ba173cc1" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
W20240503 08:12:52.572283 4770 leader_election.cc:341] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:52.572723 7742 raft_consensus.cc:2781] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:52.572808 7742 raft_consensus.cc:482] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:52.572890 7742 raft_consensus.cc:3037] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.590860 1670 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:52.587877 7742 raft_consensus.cc:504] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:52.602303 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241f60a993f94046b870eae6ba173cc1" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:52.602558 4965 raft_consensus.cc:3037] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.607841 4965 raft_consensus.cc:2454] T 241f60a993f94046b870eae6ba173cc1 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:52.609695 4768 leader_election.cc:302] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:52.610446 7607 raft_consensus.cc:2781] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:52.610631 7607 raft_consensus.cc:686] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:52.610911 7607 consensus_queue.cc:237] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:52.620923 7742 leader_election.cc:288] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:52.620981 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "241f60a993f94046b870eae6ba173cc1" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
W20240503 08:12:52.621567 4770 leader_election.cc:341] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:52.631808 7891 tablet_bootstrap.cc:492] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:52.641707 7898 tablet_bootstrap.cc:492] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:52.647661 7902 ts_tablet_manager.cc:902] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:52.648123 7851 tablet.cc:1722] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:52.649202 7868 tablet_bootstrap.cc:492] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:52.649787 7868 tablet_bootstrap.cc:492] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:12:52.650077 7868 ts_tablet_manager.cc:1367] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.265s user 0.067s sys 0.131s
I20240503 08:12:52.650480 7894 tablet_bootstrap.cc:492] T 241f60a993f94046b870eae6ba173cc1 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:52.652438 7901 ts_tablet_manager.cc:902] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:52.654431 7898 tablet_bootstrap.cc:654] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.657343 7891 tablet_bootstrap.cc:654] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.661639 1665 catalog_manager.cc:5484] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:52.676175 7854 tablet.cc:1722] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:52.678360 7868 raft_consensus.cc:348] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:52.679603 7868 raft_consensus.cc:729] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:12:52.679867 7868 consensus_queue.cc:260] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:52.682516 7868 ts_tablet_manager.cc:1398] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.032s user 0.003s sys 0.000s
I20240503 08:12:52.689057 7853 tablet.cc:1722] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:52.691824 7909 ts_tablet_manager.cc:902] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:52.691851 7873 tablet_copy_client.cc:308] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.197:42465
I20240503 08:12:52.698228 4589 tablet_copy_service.cc:140] P 5f95b05d111e4090bbc59dec394ed0ab: Received BeginTabletCopySession request for tablet b48714749469474886f9e1d7e7635915 from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:40345)
I20240503 08:12:52.698443 4589 tablet_copy_service.cc:161] P 5f95b05d111e4090bbc59dec394ed0ab: Beginning new tablet copy session on tablet b48714749469474886f9e1d7e7635915 from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:40345: session id = 184c543cb05d49cc98ba1c56617be7a5-b48714749469474886f9e1d7e7635915
I20240503 08:12:52.687906 4723 tablet_copy_service.cc:342] P 31f41542452b4b1eb994ff4223d039fc: Request end of tablet copy session b8f581390da449ccb9e299011296405e-53ce7a6567274050a2b377a9c650ccc4 received from {username='slave'} at 127.27.157.193:41709
I20240503 08:12:52.699900 7891 tablet_bootstrap.cc:492] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:52.700086 7891 ts_tablet_manager.cc:1367] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.003s
I20240503 08:12:52.701328 7891 raft_consensus.cc:348] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:52.701571 7891 raft_consensus.cc:374] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.701673 7891 raft_consensus.cc:729] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.701951 7891 consensus_queue.cc:260] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:52.702270 4589 tablet_copy_source_session.cc:205] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:52.702605 7891 ts_tablet_manager.cc:1398] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:52.703771 7898 tablet_bootstrap.cc:492] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:52.704023 7898 ts_tablet_manager.cc:1367] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.012s
I20240503 08:12:52.704793 7898 raft_consensus.cc:348] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:52.705163 7898 raft_consensus.cc:374] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.705286 7898 raft_consensus.cc:729] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.705549 7898 consensus_queue.cc:260] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:52.707620 7898 ts_tablet_manager.cc:1398] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:52.710798 7873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48714749469474886f9e1d7e7635915. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.716166 7912 ts_tablet_manager.cc:902] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:52.732538 1667 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=8680c4f223ae4b388eaaa493392d7380] alter complete (version 3)
I20240503 08:12:52.733186 7873 tablet_copy_client.cc:783] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:52.733570 7873 tablet_copy_client.cc:647] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:52.740067 7873 tablet_copy_client.cc:523] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:52.748816 7901 tablet_copy_client.cc:308] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.198:46559
I20240503 08:12:52.754635 7873 tablet_bootstrap.cc:492] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:52.757251 7912 tablet_copy_client.cc:308] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:12:52.758092 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet 34b7fd8dc7434b8ea14b9deb0023b4cb from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:35539)
I20240503 08:12:52.758282 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet 34b7fd8dc7434b8ea14b9deb0023b4cb from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:35539: session id = b8f581390da449ccb9e299011296405e-34b7fd8dc7434b8ea14b9deb0023b4cb
I20240503 08:12:52.761394 4988 tablet_copy_source_session.cc:205] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:52.763638 7912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b7fd8dc7434b8ea14b9deb0023b4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.764974 7909 tablet_copy_client.cc:308] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:41181
I20240503 08:12:52.766191 1833 tablet_copy_service.cc:140] P b8f581390da449ccb9e299011296405e: Received BeginTabletCopySession request for tablet 7d1e2530d5f2431baf0b2dd41081bb9d from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:40639)
I20240503 08:12:52.766399 1833 tablet_copy_service.cc:161] P b8f581390da449ccb9e299011296405e: Beginning new tablet copy session on tablet 7d1e2530d5f2431baf0b2dd41081bb9d from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:40639: session id = 184c543cb05d49cc98ba1c56617be7a5-7d1e2530d5f2431baf0b2dd41081bb9d
I20240503 08:12:52.770910 1833 tablet_copy_source_session.cc:205] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:52.772312 4723 tablet_copy_service.cc:433] P 31f41542452b4b1eb994ff4223d039fc: ending tablet copy session b8f581390da449ccb9e299011296405e-53ce7a6567274050a2b377a9c650ccc4 on tablet 53ce7a6567274050a2b377a9c650ccc4 with peer b8f581390da449ccb9e299011296405e
I20240503 08:12:52.774730 7909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1e2530d5f2431baf0b2dd41081bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.777670 7912 tablet_copy_client.cc:783] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:52.779654 7912 tablet_copy_client.cc:647] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:52.781287 7912 tablet_copy_client.cc:523] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:52.781338 7909 tablet_copy_client.cc:783] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:52.781602 7909 tablet_copy_client.cc:647] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:52.784849 7912 tablet_bootstrap.cc:492] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:52.789724 4723 tablet_copy_service.cc:140] P 31f41542452b4b1eb994ff4223d039fc: Received BeginTabletCopySession request for tablet 40476d1762154f6d9ce4855872492573 from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:49041)
I20240503 08:12:52.790658 4723 tablet_copy_service.cc:161] P 31f41542452b4b1eb994ff4223d039fc: Beginning new tablet copy session on tablet 40476d1762154f6d9ce4855872492573 from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:49041: session id = f24648c92c384f1ba349eee801f985e2-40476d1762154f6d9ce4855872492573
I20240503 08:12:52.792171 4700 consensus_queue.cc:237] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:52.794554 7894 tablet_bootstrap.cc:654] T 241f60a993f94046b870eae6ba173cc1 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.795300 7909 tablet_copy_client.cc:523] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:52.798998 1810 raft_consensus.cc:1203] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:52.802139 4834 raft_consensus.cc:1261] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.804651 7873 tablet_bootstrap.cc:492] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:52.805545 7873 tablet_bootstrap.cc:492] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:12:52.807238 7873 ts_tablet_manager.cc:1367] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.012s
I20240503 08:12:52.797870 4723 tablet_copy_source_session.cc:205] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:52.802888 7179 consensus_queue.cc:1035] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:52.808444 7912 tablet_bootstrap.cc:492] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:52.808861 7912 tablet_bootstrap.cc:492] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:12:52.809092 7912 ts_tablet_manager.cc:1367] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20240503 08:12:52.809985 7909 tablet_bootstrap.cc:492] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
W20240503 08:12:52.810201 4637 consensus_peers.cc:485] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 275fc5bbb1494306b1139b7f950ef80f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:52.811677 7912 raft_consensus.cc:348] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:52.811909 7912 raft_consensus.cc:729] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:12:52.812633 7901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40476d1762154f6d9ce4855872492573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.813393 7873 raft_consensus.cc:348] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:52.813768 7873 raft_consensus.cc:729] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:12:52.814028 7873 consensus_queue.cc:260] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:52.814826 7873 ts_tablet_manager.cc:1398] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:52.815991 4589 tablet_copy_service.cc:342] P 5f95b05d111e4090bbc59dec394ed0ab: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-b48714749469474886f9e1d7e7635915 received from {username='slave'} at 127.27.157.196:40345
I20240503 08:12:52.816167 4589 tablet_copy_service.cc:433] P 5f95b05d111e4090bbc59dec394ed0ab: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-b48714749469474886f9e1d7e7635915 on tablet b48714749469474886f9e1d7e7635915 with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:52.820861 7901 tablet_copy_client.cc:783] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:52.821281 7901 tablet_copy_client.cc:647] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:52.822139 7912 consensus_queue.cc:260] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:52.822883 7912 ts_tablet_manager.cc:1398] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240503 08:12:52.827602 7901 tablet_copy_client.cc:523] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:52.833045 7901 tablet_bootstrap.cc:492] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:52.833324 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session b8f581390da449ccb9e299011296405e-34b7fd8dc7434b8ea14b9deb0023b4cb received from {username='slave'} at 127.27.157.193:35539
I20240503 08:12:52.833498 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session b8f581390da449ccb9e299011296405e-34b7fd8dc7434b8ea14b9deb0023b4cb on tablet 34b7fd8dc7434b8ea14b9deb0023b4cb with peer b8f581390da449ccb9e299011296405e
I20240503 08:12:52.838987 1667 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:52.839265 1667 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:52.846771 1667 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 003c33028f594c788b84508f85a180c9
I20240503 08:12:52.848884 7876 raft_consensus.cc:482] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:52.849088 7876 raft_consensus.cc:504] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:52.848821 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c33028f594c788b84508f85a180c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:52 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:52.850142 7876 leader_election.cc:288] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:52.848181 7179 raft_consensus.cc:2932] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:52.851035 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e0415e028b4303bcf4ab64f4ec6b54" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:52.851263 1943 raft_consensus.cc:2454] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:52.851790 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e0415e028b4303bcf4ab64f4ec6b54" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:52.852020 2073 raft_consensus.cc:2454] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:52.852126 4945 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c33028f594c788b84508f85a180c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:52 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:52.852666 4503 leader_election.cc:302] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:52.853087 7876 raft_consensus.cc:2781] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:52.853193 7876 raft_consensus.cc:482] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:52.853309 7876 raft_consensus.cc:3037] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.859979 7876 raft_consensus.cc:504] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:52.861526 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e0415e028b4303bcf4ab64f4ec6b54" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:52.861825 2073 raft_consensus.cc:3037] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:12:52.864459 2010 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.199:39455, user_credentials={real_user=slave}} blocked reactor thread for 103110us
I20240503 08:12:52.866567 2073 raft_consensus.cc:2454] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:52.867257 4502 leader_election.cc:302] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:52.868002 7759 raft_consensus.cc:2781] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:52.868189 7759 raft_consensus.cc:686] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:52.868460 7759 consensus_queue.cc:237] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:52.869189 7934 tablet_replica.cc:330] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:52.869467 7934 raft_consensus.cc:2229] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:52.869370 4834 raft_consensus.cc:2932] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:52.869771 7934 raft_consensus.cc:2258] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:52.870949 7934 ts_tablet_manager.cc:1875] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 08:12:52.873212 2140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.198:46559, user_credentials={real_user=slave}} blocked reactor thread for 91158.1us
I20240503 08:12:52.874202 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 003c33028f594c788b84508f85a180c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:52 UTC) from {username='slave'} at 127.0.0.1:42570
I20240503 08:12:52.874259 7894 tablet_bootstrap.cc:492] T 241f60a993f94046b870eae6ba173cc1 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:52.874418 7894 ts_tablet_manager.cc:1367] T 241f60a993f94046b870eae6ba173cc1 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.238s user 0.003s sys 0.006s
I20240503 08:12:52.874974 2204 raft_consensus.cc:1261] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:52.875303 7876 leader_election.cc:288] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:52.875633 7894 raft_consensus.cc:348] T 241f60a993f94046b870eae6ba173cc1 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:52.875939 7894 raft_consensus.cc:374] T 241f60a993f94046b870eae6ba173cc1 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:52.875886 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e0415e028b4303bcf4ab64f4ec6b54" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:52.876096 1943 raft_consensus.cc:3037] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.876283 7894 raft_consensus.cc:729] T 241f60a993f94046b870eae6ba173cc1 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:52.876726 7894 consensus_queue.cc:260] T 241f60a993f94046b870eae6ba173cc1 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:52.878904 1943 raft_consensus.cc:2454] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:52.878331 1668 catalog_manager.cc:5484] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:52.887992 6375 raft_consensus.cc:482] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:52.888216 6375 raft_consensus.cc:504] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:52.887970 1665 catalog_manager.cc:5484] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:52.889281 6375 leader_election.cc:288] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:52.889920 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ebf60954ed44e29b5f8047ed511d2c7" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:52.890213 4966 raft_consensus.cc:2454] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:52.890305 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ebf60954ed44e29b5f8047ed511d2c7" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:52.890520 2204 raft_consensus.cc:2454] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:52.891047 1878 leader_election.cc:302] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:52.891860 6375 raft_consensus.cc:2781] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:52.891964 6375 raft_consensus.cc:482] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:52.892028 6375 raft_consensus.cc:3037] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.893700 7894 ts_tablet_manager.cc:1398] T 241f60a993f94046b870eae6ba173cc1 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240503 08:12:52.883004 7502 consensus_queue.cc:1035] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:52.882045 7901 tablet_bootstrap.cc:492] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:52.898325 7901 tablet_bootstrap.cc:492] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:12:52.898669 7901 ts_tablet_manager.cc:1367] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.005s
I20240503 08:12:52.899662 7901 raft_consensus.cc:348] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:52.899921 7901 raft_consensus.cc:729] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:12:52.900046 7902 tablet_copy_client.cc:308] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:12:52.900763 7901 consensus_queue.cc:260] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:52.904902 7901 ts_tablet_manager.cc:1398] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:12:52.905890 4723 tablet_copy_service.cc:342] P 31f41542452b4b1eb994ff4223d039fc: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-40476d1762154f6d9ce4855872492573 received from {username='slave'} at 127.27.157.199:49041
I20240503 08:12:52.906100 4723 tablet_copy_service.cc:433] P 31f41542452b4b1eb994ff4223d039fc: ending tablet copy session f24648c92c384f1ba349eee801f985e2-40476d1762154f6d9ce4855872492573 on tablet 40476d1762154f6d9ce4855872492573 with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:12:52.909338 7909 tablet_bootstrap.cc:492] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:52.909837 7909 tablet_bootstrap.cc:492] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:12:52.910145 7909 ts_tablet_manager.cc:1367] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.018s
I20240503 08:12:52.917274 7937 tablet_replica.cc:330] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:52.917632 7937 raft_consensus.cc:2229] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:52.918012 7937 raft_consensus.cc:2258] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:52.921789 7937 ts_tablet_manager.cc:1875] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:52.923002 6375 raft_consensus.cc:504] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:52.924468 4834 raft_consensus.cc:1203] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:52.911149 7909 raft_consensus.cc:348] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:52.925004 7909 raft_consensus.cc:729] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:12:52.925194 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ebf60954ed44e29b5f8047ed511d2c7" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:52.925467 4966 raft_consensus.cc:3037] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.926415 7909 consensus_queue.cc:260] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:52.931591 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet 54fe406b5f324d629243f7f939535620 from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:37935)
I20240503 08:12:52.931777 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet 54fe406b5f324d629243f7f939535620 from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:37935: session id = 31f41542452b4b1eb994ff4223d039fc-54fe406b5f324d629243f7f939535620
I20240503 08:12:52.932312 7909 ts_tablet_manager.cc:1398] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 08:12:52.933483 7932 tablet_replica.cc:330] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:52.933707 7932 raft_consensus.cc:2229] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:52.934094 7932 raft_consensus.cc:2258] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:52.934671 4855 tablet_copy_source_session.cc:205] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:52.935695 7932 ts_tablet_manager.cc:1875] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:52.936100 7902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54fe406b5f324d629243f7f939535620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.938064 1833 tablet_copy_service.cc:342] P b8f581390da449ccb9e299011296405e: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-7d1e2530d5f2431baf0b2dd41081bb9d received from {username='slave'} at 127.27.157.196:40639
I20240503 08:12:52.938217 1833 tablet_copy_service.cc:433] P b8f581390da449ccb9e299011296405e: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-7d1e2530d5f2431baf0b2dd41081bb9d on tablet 7d1e2530d5f2431baf0b2dd41081bb9d with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:52.938202 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ebf60954ed44e29b5f8047ed511d2c7" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:52.938427 2202 raft_consensus.cc:3037] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:52.941406 2202 raft_consensus.cc:2454] T 3ebf60954ed44e29b5f8047ed511d2c7 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:52.941747 4966 raft_consensus.cc:2454] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:52.942334 1878 leader_election.cc:302] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:52.944876 2204 raft_consensus.cc:2932] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:52.945642 7937 ts_tablet_manager.cc:1888] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:52.946954 7932 ts_tablet_manager.cc:1888] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:52.947141 7932 log.cc:1196] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/003c33028f594c788b84508f85a180c9
I20240503 08:12:52.948769 7937 log.cc:1196] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/003c33028f594c788b84508f85a180c9
I20240503 08:12:52.948833 7932 ts_tablet_manager.cc:1909] T 003c33028f594c788b84508f85a180c9 P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:52.949548 7937 ts_tablet_manager.cc:1909] T 003c33028f594c788b84508f85a180c9 P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:52.950700 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 003c33028f594c788b84508f85a180c9 (table rebalancer_dynamic_table_new_name [id=8680c4f223ae4b388eaaa493392d7380]) successfully deleted
I20240503 08:12:52.951573 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:52.958593 6375 leader_election.cc:288] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:52.959738 6375 raft_consensus.cc:2781] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:52.959908 6375 raft_consensus.cc:686] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:52.960358 6375 consensus_queue.cc:237] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:52.961480 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 7a95fc23399441dea4f2dd77afff610a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=39c11d13de3048e4b31b41e696057b7c]), partition=
I20240503 08:12:52.961833 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 7a95fc23399441dea4f2dd77afff610a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=39c11d13de3048e4b31b41e696057b7c]), partition=
I20240503 08:12:52.962289 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a95fc23399441dea4f2dd77afff610a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.964207 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 7a95fc23399441dea4f2dd77afff610a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=39c11d13de3048e4b31b41e696057b7c]), partition=
I20240503 08:12:52.964648 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a95fc23399441dea4f2dd77afff610a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.967185 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 003c33028f594c788b84508f85a180c9 (table rebalancer_dynamic_table_new_name [id=8680c4f223ae4b388eaaa493392d7380]) successfully deleted
I20240503 08:12:52.968482 7902 tablet_copy_client.cc:783] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:52.969030 7902 tablet_copy_client.cc:647] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:52.972357 7934 ts_tablet_manager.cc:1888] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:52.973034 7934 log.cc:1196] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/003c33028f594c788b84508f85a180c9
I20240503 08:12:52.973160 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a95fc23399441dea4f2dd77afff610a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:52.973981 7934 ts_tablet_manager.cc:1909] T 003c33028f594c788b84508f85a180c9 P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:52.975854 7882 tablet_bootstrap.cc:492] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:52.977464 7882 tablet_bootstrap.cc:654] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.985340 1670 catalog_manager.cc:5484] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:52.987900 7885 tablet_bootstrap.cc:492] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:52.989596 7902 tablet_copy_client.cc:523] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:52.990123 7885 tablet_bootstrap.cc:654] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:52.991125 7883 tablet_bootstrap.cc:492] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:53.003427 7902 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:53.005277 7883 tablet_bootstrap.cc:654] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.009699 7882 tablet_bootstrap.cc:492] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:53.009970 7882 ts_tablet_manager.cc:1367] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.006s
I20240503 08:12:53.011039 7882 raft_consensus.cc:348] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:53.015609 7882 raft_consensus.cc:374] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.015761 7882 raft_consensus.cc:729] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.018819 7882 consensus_queue.cc:260] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:53.019558 7882 ts_tablet_manager.cc:1398] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240503 08:12:53.020848 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 003c33028f594c788b84508f85a180c9 (table rebalancer_dynamic_table_new_name [id=8680c4f223ae4b388eaaa493392d7380]) successfully deleted
I20240503 08:12:53.027575 7902 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:53.028347 7902 tablet_bootstrap.cc:492] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:12:53.028718 7902 ts_tablet_manager.cc:1367] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.005s
I20240503 08:12:53.029450 7947 ts_tablet_manager.cc:902] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a: Initiating tablet copy from peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:53.030957 7883 tablet_bootstrap.cc:492] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:53.031657 7883 ts_tablet_manager.cc:1367] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.001s
I20240503 08:12:53.029891 7902 raft_consensus.cc:348] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:53.032737 7902 raft_consensus.cc:729] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:12:53.032991 7902 consensus_queue.cc:260] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:53.032593 7883 raft_consensus.cc:348] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:53.033767 7902 ts_tablet_manager.cc:1398] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:53.033923 7883 raft_consensus.cc:374] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.034601 7883 raft_consensus.cc:729] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.034904 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-54fe406b5f324d629243f7f939535620 received from {username='slave'} at 127.27.157.198:37935
I20240503 08:12:53.034885 7883 consensus_queue.cc:260] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:53.035073 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-54fe406b5f324d629243f7f939535620 on tablet 54fe406b5f324d629243f7f939535620 with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:53.035573 7883 ts_tablet_manager.cc:1398] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:53.049072 7885 tablet_bootstrap.cc:492] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:53.049379 7885 ts_tablet_manager.cc:1367] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.002s
I20240503 08:12:53.051194 7885 raft_consensus.cc:348] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:53.051609 7885 raft_consensus.cc:374] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.051730 7885 raft_consensus.cc:729] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.052023 7885 consensus_queue.cc:260] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:53.054273 7885 ts_tablet_manager.cc:1398] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:53.075856 1943 consensus_queue.cc:237] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
W20240503 08:12:53.082150 1878 consensus_peers.cc:485] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47dde43816964e89bfb235aad34e9583. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:53.084162 4566 raft_consensus.cc:1261] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:53.084756 4700 raft_consensus.cc:1261] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:53.085543 7941 consensus_queue.cc:1035] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:53.088485 7941 consensus_queue.cc:1035] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:53.107122 4698 raft_consensus.cc:3037] T 241f60a993f94046b870eae6ba173cc1 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.115664 7606 consensus_queue.cc:1035] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.131243 7941 raft_consensus.cc:2932] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:53.145828 4566 raft_consensus.cc:2932] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:53.167347 7606 consensus_queue.cc:1035] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.201962 2204 raft_consensus.cc:1203] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:53.216521 4698 raft_consensus.cc:2932] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:53.217389 1665 catalog_manager.cc:5484] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:53.246436 2204 raft_consensus.cc:1203] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:53.251269 7759 consensus_queue.cc:1035] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.263900 7607 raft_consensus.cc:482] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:53.264097 7607 raft_consensus.cc:504] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:53.266000 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a95fc23399441dea4f2dd77afff610a" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:53.266275 2204 raft_consensus.cc:2454] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:53.266726 7607 leader_election.cc:288] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:53.267138 4966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a95fc23399441dea4f2dd77afff610a" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:53.267436 4966 raft_consensus.cc:2454] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:53.267773 4768 leader_election.cc:302] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:53.268088 7606 raft_consensus.cc:2781] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:53.268222 7606 raft_consensus.cc:482] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:53.268328 7606 raft_consensus.cc:3037] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.280455 7606 raft_consensus.cc:504] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:53.281731 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a95fc23399441dea4f2dd77afff610a" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:53.281981 4965 raft_consensus.cc:3037] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.285228 4965 raft_consensus.cc:2454] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:53.286923 4768 leader_election.cc:302] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:53.288865 7606 leader_election.cc:288] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:53.289594 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a95fc23399441dea4f2dd77afff610a" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:53.289842 2202 raft_consensus.cc:3037] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.293493 2202 raft_consensus.cc:2454] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:53.294575 7606 raft_consensus.cc:2781] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:53.294848 7606 raft_consensus.cc:686] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:53.296964 7606 consensus_queue.cc:237] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:53.308740 7179 raft_consensus.cc:1050] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:12:53.309974 7179 consensus_queue.cc:237] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:53.317518 4834 raft_consensus.cc:1261] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.318614 7927 consensus_queue.cc:1035] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.324347 1668 catalog_manager.cc:5484] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:53.329597 1810 raft_consensus.cc:1203] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:53.330757 7759 consensus_queue.cc:1035] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.331636 4966 raft_consensus.cc:1261] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.332312 7927 consensus_queue.cc:1035] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.345018 2073 raft_consensus.cc:1261] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.345662 7179 consensus_queue.cc:1035] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.347642 7940 raft_consensus.cc:1050] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:12:53.348276 7940 consensus_queue.cc:237] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:12:53.354525 7947 tablet_copy_client.cc:308] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.197:42465
I20240503 08:12:53.355384 4589 tablet_copy_service.cc:140] P 5f95b05d111e4090bbc59dec394ed0ab: Received BeginTabletCopySession request for tablet 84831077defe434c9f29eecfae7b44c8 from peer 43e2e01cdd314f9c872c4336ef74250a ({username='slave'} at 127.27.157.195:44181)
I20240503 08:12:53.355568 4589 tablet_copy_service.cc:161] P 5f95b05d111e4090bbc59dec394ed0ab: Beginning new tablet copy session on tablet 84831077defe434c9f29eecfae7b44c8 from peer 43e2e01cdd314f9c872c4336ef74250a at {username='slave'} at 127.27.157.195:44181: session id = 43e2e01cdd314f9c872c4336ef74250a-84831077defe434c9f29eecfae7b44c8
I20240503 08:12:53.358827 4589 tablet_copy_source_session.cc:205] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:53.360662 7947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84831077defe434c9f29eecfae7b44c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.370594 7502 raft_consensus.cc:2932] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:53.375236 2072 raft_consensus.cc:2932] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:53.378026 4834 raft_consensus.cc:2932] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:53.383728 1668 catalog_manager.cc:5484] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 2 to 3, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:53.389619 1810 raft_consensus.cc:1261] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.397398 4966 raft_consensus.cc:2932] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:53.401854 7947 tablet_copy_client.cc:783] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:53.402213 7947 tablet_copy_client.cc:647] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:53.403846 4698 consensus_queue.cc:237] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:53.404994 7947 tablet_copy_client.cc:523] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:53.406042 1941 consensus_queue.cc:237] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:53.408018 2073 raft_consensus.cc:1261] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:53.408623 7964 consensus_queue.cc:1035] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.409152 7983 consensus_queue.cc:1035] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.409381 7947 tablet_bootstrap.cc:492] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:53.415673 7179 consensus_queue.cc:1035] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:53.416193 4834 raft_consensus.cc:1261] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:53.416911 7992 consensus_queue.cc:1035] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:53.421706 4835 raft_consensus.cc:1261] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.421581 4834 raft_consensus.cc:2932] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:53.425817 7927 consensus_queue.cc:1035] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.430596 4566 raft_consensus.cc:1261] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.431007 1670 catalog_manager.cc:5484] T 40476d1762154f6d9ce4855872492573 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:53.431473 7983 consensus_queue.cc:1035] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.436147 4945 tablet_service.cc:1514] Processing DeleteTablet for tablet 40476d1762154f6d9ce4855872492573 with delete_type TABLET_DATA_TOMBSTONED (TS 00b61625d8a7460892c118a2570afda5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:53.436599 7934 tablet_replica.cc:330] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:53.436848 7934 raft_consensus.cc:2229] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:53.437077 7934 raft_consensus.cc:2258] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:53.437212 7989 ts_tablet_manager.cc:902] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Initiating tablet copy from peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:53.440472 6375 consensus_queue.cc:1035] T 3ebf60954ed44e29b5f8047ed511d2c7 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.442239 7934 ts_tablet_manager.cc:1875] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:53.445897 7179 raft_consensus.cc:2932] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.449246 1670 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:53.451234 2072 raft_consensus.cc:2932] T 40476d1762154f6d9ce4855872492573 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:53.448483 7992 raft_consensus.cc:2932] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:53.462833 4965 raft_consensus.cc:1261] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:53.464109 7606 consensus_queue.cc:1035] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.464274 2202 raft_consensus.cc:1261] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:53.465216 7607 consensus_queue.cc:1035] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:53.466115 4834 raft_consensus.cc:2932] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.466946 4568 raft_consensus.cc:2932] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.472985 7947 tablet_bootstrap.cc:492] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:53.473459 7947 tablet_bootstrap.cc:492] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap complete.
I20240503 08:12:53.473757 7947 ts_tablet_manager.cc:1367] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.002s
I20240503 08:12:53.474550 7947 raft_consensus.cc:348] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:53.476085 4967 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:12:53.476661 7964 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:53.477303 1670 catalog_manager.cc:5484] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
W20240503 08:12:53.480337 1877 consensus_peers.cc:485] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53197deeb8d7439c90b019a6a90857e0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:53.480621 7947 raft_consensus.cc:729] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: LEARNER
I20240503 08:12:53.481353 7947 consensus_queue.cc:260] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:53.481581 7964 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:53.483449 7934 ts_tablet_manager.cc:1888] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:53.483603 7934 log.cc:1196] T 40476d1762154f6d9ce4855872492573 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/40476d1762154f6d9ce4855872492573
I20240503 08:12:53.484462 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 40476d1762154f6d9ce4855872492573 (table rebalancer_extra_table_30 [id=b6256afd3f1a487c896a8ef180a92b10]) successfully deleted
I20240503 08:12:53.484133 1810 raft_consensus.cc:2932] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.486958 7947 ts_tablet_manager.cc:1398] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20240503 08:12:53.490530 4589 tablet_copy_service.cc:342] P 5f95b05d111e4090bbc59dec394ed0ab: Request end of tablet copy session 43e2e01cdd314f9c872c4336ef74250a-84831077defe434c9f29eecfae7b44c8 received from {username='slave'} at 127.27.157.195:44181
I20240503 08:12:53.490231 4967 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:53.490725 4589 tablet_copy_service.cc:433] P 5f95b05d111e4090bbc59dec394ed0ab: ending tablet copy session 43e2e01cdd314f9c872c4336ef74250a-84831077defe434c9f29eecfae7b44c8 on tablet 84831077defe434c9f29eecfae7b44c8 with peer 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:12:53.504544 1813 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:12:53.506371 6375 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:53.509763 1665 catalog_manager.cc:5484] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 6 to 7, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:53.512984 8002 tablet.cc:1722] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:53.516647 8002 tablet.cc:1620] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:53.517095 1672 catalog_manager.cc:5576] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455) does not have the latest schema for tablet 7a95fc23399441dea4f2dd77afff610a (table rebalancer_dynamic_table [id=39c11d13de3048e4b31b41e696057b7c]). Expected version 1 got 0
I20240503 08:12:53.521150 2072 raft_consensus.cc:1203] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:53.538769 7989 tablet_copy_client.cc:308] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.198:46559
I20240503 08:12:53.539378 4698 consensus_queue.cc:237] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:12:53.539775 4723 tablet_copy_service.cc:140] P 31f41542452b4b1eb994ff4223d039fc: Received BeginTabletCopySession request for tablet 275fc5bbb1494306b1139b7f950ef80f from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a ({username='slave'} at 127.27.157.194:52331)
I20240503 08:12:53.539932 4723 tablet_copy_service.cc:161] P 31f41542452b4b1eb994ff4223d039fc: Beginning new tablet copy session on tablet 275fc5bbb1494306b1139b7f950ef80f from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a at {username='slave'} at 127.27.157.194:52331: session id = 315f3f0bcbf447a4b93b1aafcfbf0f5a-275fc5bbb1494306b1139b7f950ef80f
I20240503 08:12:53.544591 1813 raft_consensus.cc:1261] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:53.545249 7179 consensus_queue.cc:1035] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:53.546063 4723 tablet_copy_source_session.cc:205] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:53.547765 7989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275fc5bbb1494306b1139b7f950ef80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.550220 7179 raft_consensus.cc:2932] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.551857 1813 raft_consensus.cc:2932] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.558815 1668 catalog_manager.cc:5484] T 53ce7a6567274050a2b377a9c650ccc4 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.559180 4835 raft_consensus.cc:1261] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:53.560201 7983 consensus_queue.cc:1035] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:53.562937 4698 raft_consensus.cc:1203] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:12:53.563398 4835 raft_consensus.cc:2932] T 53ce7a6567274050a2b377a9c650ccc4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.565200 1667 catalog_manager.cc:5576] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183) does not have the latest schema for tablet 7a95fc23399441dea4f2dd77afff610a (table rebalancer_dynamic_table [id=39c11d13de3048e4b31b41e696057b7c]). Expected version 1 got 0
I20240503 08:12:53.569514 4547 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ce7a6567274050a2b377a9c650ccc4 with delete_type TABLET_DATA_TOMBSTONED (TS 5f95b05d111e4090bbc59dec394ed0ab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:53.581014 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53ce7a6567274050a2b377a9c650ccc4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:53.583462 8009 tablet.cc:1722] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:53.584229 8009 tablet.cc:1620] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: MemRowSet was empty: no flush needed.
I20240503 08:12:53.589293 1668 catalog_manager.cc:5576] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183) does not have the latest schema for tablet 7a95fc23399441dea4f2dd77afff610a (table rebalancer_dynamic_table [id=39c11d13de3048e4b31b41e696057b7c]). Expected version 1 got 0
I20240503 08:12:53.597193 7989 tablet_copy_client.cc:783] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:53.597617 7989 tablet_copy_client.cc:647] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:53.597517 1667 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:53.597144 1670 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=39c11d13de3048e4b31b41e696057b7c] alter complete (version 1)
I20240503 08:12:53.607409 7989 tablet_copy_client.cc:523] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:53.612206 8008 tablet_replica.cc:330] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:53.612466 8008 raft_consensus.cc:2229] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:53.612893 8008 raft_consensus.cc:2258] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:53.613528 8007 ts_tablet_manager.cc:902] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:53.615971 8003 tablet.cc:1722] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:53.616727 8003 tablet.cc:1620] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:53.619545 1663 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 7a95fc23399441dea4f2dd77afff610a (table rebalancer_dynamic_table [id=39c11d13de3048e4b31b41e696057b7c]). Expected version 1 got 0
I20240503 08:12:53.621556 1810 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:12:53.624475 7989 tablet_bootstrap.cc:492] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:53.630981 4547 tablet_service.cc:1467] Processing CreateTablet for tablet 7a792a309b3c43f482b4adb0fbcfadde (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=682625ce97fd4bbaa5012a004977db42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.631271 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 7a792a309b3c43f482b4adb0fbcfadde (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=682625ce97fd4bbaa5012a004977db42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.631659 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a792a309b3c43f482b4adb0fbcfadde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.631788 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a792a309b3c43f482b4adb0fbcfadde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.635735 8008 ts_tablet_manager.cc:1875] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:53.636592 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40476d1762154f6d9ce4855872492573 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:53.637130 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 7a792a309b3c43f482b4adb0fbcfadde (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=682625ce97fd4bbaa5012a004977db42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.637631 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a792a309b3c43f482b4adb0fbcfadde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.639755 4944 tablet_service.cc:1467] Processing CreateTablet for tablet 5785af2b9ba34a3b95c5ffdb024b1c8f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=682625ce97fd4bbaa5012a004977db42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.640179 4944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5785af2b9ba34a3b95c5ffdb024b1c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.642648 4548 tablet_service.cc:1467] Processing CreateTablet for tablet a76b5650ad0b400da5db935edd1df6b9 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=682625ce97fd4bbaa5012a004977db42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.643116 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76b5650ad0b400da5db935edd1df6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.643893 4947 tablet_service.cc:1467] Processing CreateTablet for tablet a76b5650ad0b400da5db935edd1df6b9 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=682625ce97fd4bbaa5012a004977db42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.644470 4947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76b5650ad0b400da5db935edd1df6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.647687 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 5785af2b9ba34a3b95c5ffdb024b1c8f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=682625ce97fd4bbaa5012a004977db42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.648226 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5785af2b9ba34a3b95c5ffdb024b1c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.648234 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 5785af2b9ba34a3b95c5ffdb024b1c8f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=682625ce97fd4bbaa5012a004977db42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.648640 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5785af2b9ba34a3b95c5ffdb024b1c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.664973 8014 tablet_bootstrap.cc:492] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:53.666960 8014 tablet_bootstrap.cc:654] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.667965 8013 tablet_bootstrap.cc:492] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:53.669948 8013 tablet_bootstrap.cc:654] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.679015 4812 tablet_service.cc:1467] Processing CreateTablet for tablet a76b5650ad0b400da5db935edd1df6b9 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=682625ce97fd4bbaa5012a004977db42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:53.684608 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76b5650ad0b400da5db935edd1df6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:53.704890 1667 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:53.725477 1667 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 7a95fc23399441dea4f2dd77afff610a (table rebalancer_dynamic_table [id=39c11d13de3048e4b31b41e696057b7c]). Expected version 2 got 1
I20240503 08:12:53.728190 8023 tablet_bootstrap.cc:492] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:53.730566 8023 tablet_bootstrap.cc:654] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.740826 8022 tablet_bootstrap.cc:492] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:53.743252 8022 tablet_bootstrap.cc:654] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.745962 7184 raft_consensus.cc:1050] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:12:53.746912 7184 consensus_queue.cc:237] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:12:53.750406 8009 tablet.cc:1722] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:53.751158 8009 tablet.cc:1620] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: MemRowSet was empty: no flush needed.
I20240503 08:12:53.753751 8021 tablet_bootstrap.cc:492] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:53.756968 8021 tablet_bootstrap.cc:654] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.758819 8020 tablet_bootstrap.cc:492] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:53.764575 8003 tablet.cc:1722] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:53.766456 8003 tablet.cc:1620] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:53.742933 8002 tablet.cc:1722] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:53.774533 8002 tablet.cc:1620] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: MemRowSet was empty: no flush needed.
I20240503 08:12:53.768188 7989 tablet_bootstrap.cc:492] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:53.778098 7989 tablet_bootstrap.cc:492] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap complete.
I20240503 08:12:53.778415 7989 ts_tablet_manager.cc:1367] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.154s user 0.020s sys 0.014s
I20240503 08:12:53.778522 8020 tablet_bootstrap.cc:654] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.779348 7989 raft_consensus.cc:348] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:53.779618 7989 raft_consensus.cc:729] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: LEARNER
I20240503 08:12:53.779809 7989 consensus_queue.cc:260] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:53.780400 7989 ts_tablet_manager.cc:1398] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:53.780896 1941 raft_consensus.cc:1261] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.776794 1810 raft_consensus.cc:1261] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.781589 7184 consensus_queue.cc:1035] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.782052 7585 consensus_queue.cc:1035] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.785573 4723 tablet_copy_service.cc:342] P 31f41542452b4b1eb994ff4223d039fc: Request end of tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-275fc5bbb1494306b1139b7f950ef80f received from {username='slave'} at 127.27.157.194:52331
I20240503 08:12:53.785805 4723 tablet_copy_service.cc:433] P 31f41542452b4b1eb994ff4223d039fc: ending tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-275fc5bbb1494306b1139b7f950ef80f on tablet 275fc5bbb1494306b1139b7f950ef80f with peer 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:12:53.770571 7764 raft_consensus.cc:1050] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:12:53.796671 8023 tablet_bootstrap.cc:492] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:53.796830 8023 ts_tablet_manager.cc:1367] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20240503 08:12:53.797057 7764 consensus_queue.cc:237] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:53.797966 8023 raft_consensus.cc:348] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.798166 8023 raft_consensus.cc:374] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.798238 8023 raft_consensus.cc:729] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.781150 4568 raft_consensus.cc:1261] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.803653 8023 consensus_queue.cc:260] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.804488 7184 consensus_queue.cc:1035] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.805266 8023 ts_tablet_manager.cc:1398] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:12:53.806030 8014 tablet_bootstrap.cc:492] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:53.806255 8014 ts_tablet_manager.cc:1367] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.141s user 0.000s sys 0.007s
I20240503 08:12:53.807307 7585 raft_consensus.cc:2932] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.807833 8014 raft_consensus.cc:348] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:53.808055 8014 raft_consensus.cc:374] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.808123 8014 raft_consensus.cc:729] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.808430 8014 consensus_queue.cc:260] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:53.810657 8014 ts_tablet_manager.cc:1398] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:53.820379 8013 tablet_bootstrap.cc:492] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:53.820945 8013 ts_tablet_manager.cc:1367] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.172s user 0.002s sys 0.006s
I20240503 08:12:53.826637 1670 catalog_manager.cc:5484] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:53.827122 1942 raft_consensus.cc:1203] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:53.828522 8020 tablet_bootstrap.cc:492] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:53.828752 8020 ts_tablet_manager.cc:1367] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.003s
I20240503 08:12:53.823892 1941 raft_consensus.cc:2932] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.829777 8020 raft_consensus.cc:348] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.830054 8020 raft_consensus.cc:374] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.836238 8020 raft_consensus.cc:729] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.836653 8020 consensus_queue.cc:260] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.839684 8020 ts_tablet_manager.cc:1398] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 08:12:53.840052 8020 tablet_bootstrap.cc:492] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:53.841557 8020 tablet_bootstrap.cc:654] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.844552 4967 consensus_queue.cc:237] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:12:53.846150 8020 tablet_bootstrap.cc:492] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:53.847383 1941 raft_consensus.cc:1261] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:53.849504 8020 ts_tablet_manager.cc:1367] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240503 08:12:53.848136 7585 consensus_queue.cc:1035] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:53.850385 8020 raft_consensus.cc:348] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.850615 8020 raft_consensus.cc:374] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.850697 8020 raft_consensus.cc:729] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.853181 7184 raft_consensus.cc:2932] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.853822 1941 raft_consensus.cc:2932] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.856122 8020 consensus_queue.cc:260] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.856755 8020 ts_tablet_manager.cc:1398] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20240503 08:12:53.857545 2139 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.194:41975, user_credentials={real_user=slave}} blocked reactor thread for 123254us
I20240503 08:12:53.858124 8020 tablet_bootstrap.cc:492] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:53.860175 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 34b7fd8dc7434b8ea14b9deb0023b4cb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:53.862886 8008 ts_tablet_manager.cc:1888] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:53.863163 8008 log.cc:1196] T 53ce7a6567274050a2b377a9c650ccc4 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/53ce7a6567274050a2b377a9c650ccc4
I20240503 08:12:53.863749 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=39c11d13de3048e4b31b41e696057b7c] alter complete (version 2)
I20240503 08:12:53.864418 8013 raft_consensus.cc:348] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.864710 8013 raft_consensus.cc:374] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.864933 8013 raft_consensus.cc:729] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.865233 8013 consensus_queue.cc:260] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.865890 8020 tablet_bootstrap.cc:654] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.865986 8013 ts_tablet_manager.cc:1398] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.045s user 0.000s sys 0.002s
I20240503 08:12:53.867197 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 53ce7a6567274050a2b377a9c650ccc4 (table rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]) successfully deleted
I20240503 08:12:53.875664 8007 tablet_copy_client.cc:308] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:53.881373 2204 raft_consensus.cc:1261] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.881157 1670 catalog_manager.cc:5484] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:53.882124 7876 consensus_queue.cc:1035] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.883710 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 47dde43816964e89bfb235aad34e9583 from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:35673)
I20240503 08:12:53.883893 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 47dde43816964e89bfb235aad34e9583 from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:35673: session id = 184c543cb05d49cc98ba1c56617be7a5-47dde43816964e89bfb235aad34e9583
I20240503 08:12:53.885850 1941 raft_consensus.cc:1261] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.886647 7876 consensus_queue.cc:1035] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.887135 1964 tablet_copy_source_session.cc:205] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:53.814035 1810 raft_consensus.cc:2932] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.887735 4547 tablet_service.cc:1514] Processing DeleteTablet for tablet 34b7fd8dc7434b8ea14b9deb0023b4cb with delete_type TABLET_DATA_TOMBSTONED (TS 5f95b05d111e4090bbc59dec394ed0ab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:53.824007 7831 raft_consensus.cc:1050] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:12:53.891892 1812 consensus_queue.cc:237] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:53.892131 7831 consensus_queue.cc:237] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:53.892769 4703 raft_consensus.cc:1261] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.894208 8008 tablet_replica.cc:330] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:53.894552 8008 raft_consensus.cc:2229] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:53.894796 8008 pending_rounds.cc:62] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab: Trying to abort 1 pending ops.
I20240503 08:12:53.894877 8008 pending_rounds.cc:68] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023509396464906240 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "34b7fd8dc7434b8ea14b9deb0023b4cb" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } } }
I20240503 08:12:53.895490 8008 raft_consensus.cc:2894] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 08:12:53.895803 8021 tablet_bootstrap.cc:492] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:53.895965 8021 ts_tablet_manager.cc:1367] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.142s user 0.005s sys 0.000s
I20240503 08:12:53.896816 8021 raft_consensus.cc:348] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.897127 8021 raft_consensus.cc:374] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.897501 1813 raft_consensus.cc:1261] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:53.897961 8021 raft_consensus.cc:729] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.898391 7184 consensus_queue.cc:1035] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:53.898356 8021 consensus_queue.cc:260] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.898852 7759 consensus_queue.cc:1035] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.900413 1813 raft_consensus.cc:2932] T 34b7fd8dc7434b8ea14b9deb0023b4cb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:53.901139 8008 raft_consensus.cc:2866] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:12:53.901331 8008 raft_consensus.cc:2258] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:53.908149 8008 ts_tablet_manager.cc:1875] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:53.910652 2202 raft_consensus.cc:1261] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.911700 1942 raft_consensus.cc:1261] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.911600 7737 consensus_queue.cc:1035] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.912688 8021 ts_tablet_manager.cc:1398] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 08:12:53.914135 7737 consensus_queue.cc:1035] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.915091 8021 tablet_bootstrap.cc:492] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:53.920392 7184 raft_consensus.cc:482] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:53.920620 7184 raft_consensus.cc:504] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.921540 7184 leader_election.cc:288] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:53.922474 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76b5650ad0b400da5db935edd1df6b9" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:53.923524 4568 raft_consensus.cc:2454] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:53.924631 4900 leader_election.cc:302] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:53.925602 7184 raft_consensus.cc:2781] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:53.925747 7184 raft_consensus.cc:482] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:53.925845 7184 raft_consensus.cc:3037] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.928922 7184 raft_consensus.cc:504] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.930302 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76b5650ad0b400da5db935edd1df6b9" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:53.930598 4568 raft_consensus.cc:3037] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.933162 8020 tablet_bootstrap.cc:492] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:53.933364 8020 ts_tablet_manager.cc:1367] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.009s
I20240503 08:12:53.933926 4568 raft_consensus.cc:2454] T a76b5650ad0b400da5db935edd1df6b9 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:53.934268 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76b5650ad0b400da5db935edd1df6b9" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:53.934592 4835 raft_consensus.cc:2454] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:53.934455 8020 raft_consensus.cc:348] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:53.934749 8020 raft_consensus.cc:374] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.934855 8020 raft_consensus.cc:729] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.936503 8022 tablet_bootstrap.cc:492] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:53.936774 8022 ts_tablet_manager.cc:1367] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.196s user 0.005s sys 0.016s
I20240503 08:12:53.937763 8022 raft_consensus.cc:348] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:53.938421 8022 raft_consensus.cc:374] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:53.938436 4900 leader_election.cc:302] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:53.938553 8022 raft_consensus.cc:729] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:53.938839 8041 raft_consensus.cc:2781] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:53.938978 8041 raft_consensus.cc:686] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:53.939349 8041 consensus_queue.cc:237] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:53.942054 4967 raft_consensus.cc:1261] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:53.942974 8022 consensus_queue.cc:260] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:53.944052 8022 ts_tablet_manager.cc:1398] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240503 08:12:53.937464 7876 raft_consensus.cc:1050] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab: attempting to promote NON_VOTER 43e2e01cdd314f9c872c4336ef74250a to VOTER
I20240503 08:12:53.945463 7737 consensus_queue.cc:1035] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:53.945868 8021 tablet_bootstrap.cc:654] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:53.945659 7876 consensus_queue.cc:237] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:12:53.948853 7759 raft_consensus.cc:2932] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:53.948915 7737 raft_consensus.cc:2932] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:53.952903 4698 raft_consensus.cc:2932] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:53.953781 2202 raft_consensus.cc:2932] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:53.935153 8020 consensus_queue.cc:260] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:53.957274 8020 ts_tablet_manager.cc:1398] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20240503 08:12:53.961546 7742 raft_consensus.cc:1050] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:12:53.962774 7742 consensus_queue.cc:237] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:53.967479 1670 catalog_manager.cc:5484] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:53.971442 1942 raft_consensus.cc:2932] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:53.986979 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76b5650ad0b400da5db935edd1df6b9" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:53.987654 4835 raft_consensus.cc:3037] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:53.987627 1810 consensus_queue.cc:237] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:53.994946 1665 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:53.998631 7737 raft_consensus.cc:482] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:53.999917 7737 raft_consensus.cc:504] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:54.000758 2202 raft_consensus.cc:1261] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:54.002179 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5785af2b9ba34a3b95c5ffdb024b1c8f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:54.002557 1942 raft_consensus.cc:2454] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:54.002921 7737 leader_election.cc:288] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:54.003090 1748 leader_election.cc:302] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:54.003136 7737 consensus_queue.cc:1035] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:54.003425 7831 raft_consensus.cc:2781] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:54.003501 7831 raft_consensus.cc:482] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:54.003578 7831 raft_consensus.cc:3037] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.009591 7831 raft_consensus.cc:504] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:54.009960 2204 raft_consensus.cc:2932] T 7d1e2530d5f2431baf0b2dd41081bb9d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:54.018777 8002 tablet.cc:1722] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:54.024737 8021 tablet_bootstrap.cc:492] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:54.024991 8021 ts_tablet_manager.cc:1367] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.003s
I20240503 08:12:54.025893 8021 raft_consensus.cc:348] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:54.026252 8021 raft_consensus.cc:374] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.026389 8021 raft_consensus.cc:729] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.026707 8021 consensus_queue.cc:260] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:54.029536 4700 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:54.030051 2072 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:54.030247 7607 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:54.031400 7831 leader_election.cc:288] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:54.031858 2201 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:12:54.031868 7606 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:54.031704 7831 raft_consensus.cc:2932] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:54.032040 8009 tablet.cc:1722] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:53.976069 4967 raft_consensus.cc:2932] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:54.006008 4964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5785af2b9ba34a3b95c5ffdb024b1c8f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:54.033406 4964 raft_consensus.cc:2454] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:54.037093 4965 raft_consensus.cc:1261] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:54.038961 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d1e2530d5f2431baf0b2dd41081bb9d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:54.040885 4964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5785af2b9ba34a3b95c5ffdb024b1c8f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:54.041123 4964 raft_consensus.cc:3037] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.042786 8003 tablet.cc:1722] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:54.044274 2201 raft_consensus.cc:2932] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:54.048053 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5785af2b9ba34a3b95c5ffdb024b1c8f" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:54.048300 1942 raft_consensus.cc:3037] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.051079 4964 raft_consensus.cc:2454] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:54.053145 1670 catalog_manager.cc:5484] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 3 to 4, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:54.054639 1747 leader_election.cc:302] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:54.059437 7184 leader_election.cc:288] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:54.060722 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1e2530d5f2431baf0b2dd41081bb9d with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:54.066907 7737 consensus_queue.cc:1035] T 7d1e2530d5f2431baf0b2dd41081bb9d P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240503 08:12:54.070525 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=39c11d13de3048e4b31b41e696057b7c] alter complete (version 3)
I20240503 08:12:54.070784 4964 raft_consensus.cc:2932] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:54.072875 7737 raft_consensus.cc:2781] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:54.073031 7737 raft_consensus.cc:686] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:12:54.073282 7737 consensus_queue.cc:237] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:54.073817 1942 raft_consensus.cc:2454] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:12:54.079106 4835 raft_consensus.cc:2454] T a76b5650ad0b400da5db935edd1df6b9 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:54.086819 1668 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 7a95fc23399441dea4f2dd77afff610a (table rebalancer_dynamic_table_new_name [id=39c11d13de3048e4b31b41e696057b7c]). Expected version 3 got 2
I20240503 08:12:54.087661 7742 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:12:54.099346 8069 tablet_replica.cc:330] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:54.099834 7606 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:54.108284 1668 catalog_manager.cc:5484] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:54.108784 2072 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:54.109340 4700 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
W20240503 08:12:54.102244 8046 log.cc:924] Time spent T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Append to log took a long time: real 0.213s user 0.000s sys 0.000s
I20240503 08:12:54.110317 8069 raft_consensus.cc:2229] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.119202 8069 raft_consensus.cc:2258] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.120013 7992 raft_consensus.cc:482] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:54.120118 8021 ts_tablet_manager.cc:1398] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.095s user 0.001s sys 0.001s
I20240503 08:12:54.120186 7992 raft_consensus.cc:504] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:54.120957 7992 leader_election.cc:288] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:54.122280 4964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a792a309b3c43f482b4adb0fbcfadde" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:54.122426 2072 raft_consensus.cc:1261] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:54.122597 4964 raft_consensus.cc:2454] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:54.123338 4636 leader_election.cc:302] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:54.123709 7992 raft_consensus.cc:2781] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:54.123831 7992 raft_consensus.cc:482] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:54.123939 7992 raft_consensus.cc:3037] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.126932 7992 raft_consensus.cc:504] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:54.128522 2202 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:54.129390 7992 leader_election.cc:288] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:54.129547 4964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a792a309b3c43f482b4adb0fbcfadde" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:54.129752 4964 raft_consensus.cc:3037] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.132331 1665 catalog_manager.cc:5484] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 5 to 6, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:54.132839 4964 raft_consensus.cc:2454] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:54.133517 4636 leader_election.cc:302] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:54.135689 7992 raft_consensus.cc:2781] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:54.135728 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a792a309b3c43f482b4adb0fbcfadde" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:54.135972 4568 raft_consensus.cc:2454] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:54.135963 7992 raft_consensus.cc:686] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:54.136091 4566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a792a309b3c43f482b4adb0fbcfadde" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:54.136305 4566 raft_consensus.cc:3037] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.136298 7992 consensus_queue.cc:237] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:54.139113 4566 raft_consensus.cc:2454] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:54.139014 7764 consensus_queue.cc:1035] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:54.173089 8007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47dde43816964e89bfb235aad34e9583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.175812 8069 ts_tablet_manager.cc:1875] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:54.142047 1667 catalog_manager.cc:5484] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:54.185607 8007 tablet_copy_client.cc:783] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:54.172398 4835 consensus_queue.cc:237] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:54.186900 8007 tablet_copy_client.cc:647] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:54.192222 4698 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:12:54.192754 2204 raft_consensus.cc:1261] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:12:54.193102 8065 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:54.193480 7607 consensus_queue.cc:1035] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:54.197346 7607 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:54.199070 4698 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:54.205673 2204 raft_consensus.cc:2932] T 54fe406b5f324d629243f7f939535620 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:54.206475 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54fe406b5f324d629243f7f939535620 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:54.209683 1663 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:54.209941 1663 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:54.220583 1663 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 7a95fc23399441dea4f2dd77afff610a
I20240503 08:12:54.222398 4812 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a95fc23399441dea4f2dd77afff610a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:54 UTC) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:54.227196 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a95fc23399441dea4f2dd77afff610a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:54 UTC) from {username='slave'} at 127.0.0.1:42570
W20240503 08:12:54.232626 4504 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.198:46559, user_credentials={real_user=slave}} blocked reactor thread for 260176us
I20240503 08:12:54.233542 8069 ts_tablet_manager.cc:1888] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:54.235914 8069 log.cc:1196] T 7d1e2530d5f2431baf0b2dd41081bb9d P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/7d1e2530d5f2431baf0b2dd41081bb9d
I20240503 08:12:54.239388 4568 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:12:54.239969 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 7d1e2530d5f2431baf0b2dd41081bb9d (table rebalancer_extra_table_24 [id=d79878310527447b9246772f2dcc1501]) successfully deleted
I20240503 08:12:54.240168 7737 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:12:54.244735 1747 consensus_peers.cc:485] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3219159f92774adbb7ecaa77ac267021. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:54.248477 4567 consensus_queue.cc:237] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:54.250780 1942 raft_consensus.cc:2932] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:54.251614 2204 raft_consensus.cc:1261] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:54.253319 8008 ts_tablet_manager.cc:1888] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:54.253491 8008 log.cc:1196] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/34b7fd8dc7434b8ea14b9deb0023b4cb
I20240503 08:12:54.256341 8076 tablet_replica.cc:330] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:12:54.256708 8076 raft_consensus.cc:2229] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.257174 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 34b7fd8dc7434b8ea14b9deb0023b4cb (table rebalancer_extra_table_2 [id=04d15c059bf04757a1646fc42ab0184e]) successfully deleted
I20240503 08:12:54.257453 8076 raft_consensus.cc:2258] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.259066 8063 consensus_queue.cc:1035] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:54.260268 8076 ts_tablet_manager.cc:1875] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:54.262028 8059 raft_consensus.cc:2932] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:54.262447 1942 raft_consensus.cc:1261] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:54.262284 2204 raft_consensus.cc:2932] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:54.263636 8063 consensus_queue.cc:1035] T b48714749469474886f9e1d7e7635915 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:54.265233 1942 raft_consensus.cc:2932] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:54.271004 2072 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:12:54.274878 8062 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:54.283219 4967 raft_consensus.cc:1261] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:54.283421 1656 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b48714749469474886f9e1d7e7635915 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:54.283627 8007 tablet_copy_client.cc:523] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:54.284122 7764 consensus_queue.cc:1035] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:54.287274 8007 tablet_bootstrap.cc:492] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:54.290354 7737 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:54.292505 8075 tablet_replica.cc:330] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:54.292810 8075 raft_consensus.cc:2229] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:54.293464 8075 raft_consensus.cc:2258] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.294679 8075 ts_tablet_manager.cc:1875] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:54.299870 8078 ts_tablet_manager.cc:902] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:54.300675 8007 tablet_bootstrap.cc:492] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:54.301175 8007 tablet_bootstrap.cc:492] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:12:54.301982 8059 raft_consensus.cc:2932] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:54.302529 1667 catalog_manager.cc:5484] T 54fe406b5f324d629243f7f939535620 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 6 to 7, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:54.303653 7179 raft_consensus.cc:1050] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc: attempting to promote NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a to VOTER
I20240503 08:12:54.303982 4945 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a95fc23399441dea4f2dd77afff610a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:54 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:54.304561 7179 consensus_queue.cc:237] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:12:54.309136 8007 ts_tablet_manager.cc:1367] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.003s
I20240503 08:12:54.303390 1667 catalog_manager.cc:5484] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:54.310097 8007 raft_consensus.cc:348] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:54.309062 4967 raft_consensus.cc:2932] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:54.310566 8007 raft_consensus.cc:729] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:12:54.310815 8007 consensus_queue.cc:260] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:54.311643 8007 ts_tablet_manager.cc:1398] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:54.311883 4835 raft_consensus.cc:1261] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:54.315646 7759 consensus_queue.cc:1035] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:54.279592 1668 catalog_manager.cc:5484] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:54.326961 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-47dde43816964e89bfb235aad34e9583 received from {username='slave'} at 127.27.157.196:35673
I20240503 08:12:54.327104 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-47dde43816964e89bfb235aad34e9583 on tablet 47dde43816964e89bfb235aad34e9583 with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:54.329108 1672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:54.328966 1665 catalog_manager.cc:5484] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:54.332829 1666 catalog_manager.cc:5484] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:54.332893 4835 raft_consensus.cc:2932] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:54.335543 2072 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:54.342902 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet b48714749469474886f9e1d7e7635915 with delete_type TABLET_DATA_TOMBSTONED (TS 31f41542452b4b1eb994ff4223d039fc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:54.346246 8076 ts_tablet_manager.cc:1888] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:54.349738 8076 log.cc:1196] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/7a95fc23399441dea4f2dd77afff610a
I20240503 08:12:54.354156 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet 54fe406b5f324d629243f7f939535620 with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:54.354707 8076 ts_tablet_manager.cc:1909] T 7a95fc23399441dea4f2dd77afff610a P 184c543cb05d49cc98ba1c56617be7a5: Deleting consensus metadata
I20240503 08:12:54.359171 2073 raft_consensus.cc:2932] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:54.362813 1667 catalog_manager.cc:5484] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:54.366955 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 7a95fc23399441dea4f2dd77afff610a (table rebalancer_dynamic_table_new_name [id=39c11d13de3048e4b31b41e696057b7c]) successfully deleted
I20240503 08:12:54.367605 8084 tablet_replica.cc:330] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:54.368784 4568 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:12:54.370769 8084 raft_consensus.cc:2229] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.373111 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 9da67bb92e824275b0e12f97ad95e102 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b315b0aaa3124d5081e0d226641eb2da]), partition=
I20240503 08:12:54.373541 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da67bb92e824275b0e12f97ad95e102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.373503 8084 raft_consensus.cc:2258] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.373661 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 9da67bb92e824275b0e12f97ad95e102 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b315b0aaa3124d5081e0d226641eb2da]), partition=
I20240503 08:12:54.374037 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da67bb92e824275b0e12f97ad95e102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.380753 8014 tablet_bootstrap.cc:492] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:54.381053 8084 ts_tablet_manager.cc:1875] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:54.382335 8014 tablet_bootstrap.cc:654] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.385334 8089 tablet_replica.cc:330] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:54.392573 4547 tablet_service.cc:1467] Processing CreateTablet for tablet 9da67bb92e824275b0e12f97ad95e102 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b315b0aaa3124d5081e0d226641eb2da]), partition=
I20240503 08:12:54.393045 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da67bb92e824275b0e12f97ad95e102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.397668 1942 raft_consensus.cc:1261] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:54.399001 7940 consensus_queue.cc:1035] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:54.399883 8089 raft_consensus.cc:2229] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.401662 2204 raft_consensus.cc:1261] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:54.401960 8064 consensus_queue.cc:1035] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.402508 7992 consensus_queue.cc:1035] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:54.407991 8093 tablet_bootstrap.cc:492] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:54.415643 4834 raft_consensus.cc:1261] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:54.416146 8089 raft_consensus.cc:2258] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.416390 7992 consensus_queue.cc:1035] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:54.419991 8014 tablet_bootstrap.cc:492] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:54.420812 8014 ts_tablet_manager.cc:1367] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.001s
I20240503 08:12:54.422011 8014 raft_consensus.cc:348] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:54.418658 8091 tablet_replica.cc:330] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:54.422763 8091 raft_consensus.cc:2229] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.423228 8091 raft_consensus.cc:2258] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.429529 8064 consensus_queue.cc:1035] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.430119 8014 raft_consensus.cc:374] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.428123 8089 ts_tablet_manager.cc:1875] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:54.431826 8091 ts_tablet_manager.cc:1875] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:54.431226 8014 raft_consensus.cc:729] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.432577 8014 consensus_queue.cc:260] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:54.435604 8093 tablet_bootstrap.cc:654] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.441102 8084 ts_tablet_manager.cc:1888] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:54.442014 8084 log.cc:1196] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/7a95fc23399441dea4f2dd77afff610a
I20240503 08:12:54.442775 8084 ts_tablet_manager.cc:1909] T 7a95fc23399441dea4f2dd77afff610a P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:54.450997 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 7a95fc23399441dea4f2dd77afff610a (table rebalancer_dynamic_table_new_name [id=39c11d13de3048e4b31b41e696057b7c]) successfully deleted
I20240503 08:12:54.451541 8014 ts_tablet_manager.cc:1398] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240503 08:12:54.452848 8021 tablet_bootstrap.cc:492] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:54.458487 8021 tablet_bootstrap.cc:654] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.465795 8075 ts_tablet_manager.cc:1888] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:54.466135 8075 log.cc:1196] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/7a95fc23399441dea4f2dd77afff610a
I20240503 08:12:54.466780 8075 ts_tablet_manager.cc:1909] T 7a95fc23399441dea4f2dd77afff610a P f24648c92c384f1ba349eee801f985e2: Deleting consensus metadata
I20240503 08:12:54.468268 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet 7a95fc23399441dea4f2dd77afff610a (table rebalancer_dynamic_table_new_name [id=39c11d13de3048e4b31b41e696057b7c]) successfully deleted
I20240503 08:12:54.474300 7992 raft_consensus.cc:2932] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:54.475122 8050 consensus_queue.cc:1035] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.478471 4835 raft_consensus.cc:2932] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:54.485723 7737 consensus_queue.cc:1035] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.492882 8078 tablet_copy_client.cc:308] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:54.493778 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 53197deeb8d7439c90b019a6a90857e0 from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:32893)
I20240503 08:12:54.493994 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 53197deeb8d7439c90b019a6a90857e0 from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:32893: session id = 5f95b05d111e4090bbc59dec394ed0ab-53197deeb8d7439c90b019a6a90857e0
I20240503 08:12:54.498087 1964 tablet_copy_source_session.cc:205] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:54.506036 1668 catalog_manager.cc:5484] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:54.509827 1941 raft_consensus.cc:2932] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:54.520676 2202 raft_consensus.cc:1203] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:54.521895 2204 raft_consensus.cc:2932] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:54.531667 4567 consensus_queue.cc:237] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:12:54.535604 4835 raft_consensus.cc:1261] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:54.538507 2072 raft_consensus.cc:1261] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:54.539310 8059 consensus_queue.cc:1035] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20240503 08:12:54.540447 1943 consensus_queue.cc:237] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:54.542212 8063 consensus_queue.cc:1035] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240503 08:12:54.547467 8059 raft_consensus.cc:2932] T 84831077defe434c9f29eecfae7b44c8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:54.550319 2073 raft_consensus.cc:2932] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
W20240503 08:12:54.552299 1880 consensus_peers.cc:485] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cf9b2eef42e4fff8cdf5f9dd7b8b31e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:54.552150 4833 raft_consensus.cc:2932] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:54.553695 1812 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:12:54.554837 7941 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:54.555430 1656 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84831077defe434c9f29eecfae7b44c8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:54.556612 4967 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:12:54.557384 7964 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:12:54.560335 8078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53197deeb8d7439c90b019a6a90857e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.567071 8021 tablet_bootstrap.cc:492] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:54.567370 8021 ts_tablet_manager.cc:1367] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.115s user 0.012s sys 0.017s
I20240503 08:12:54.568356 8021 raft_consensus.cc:348] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:54.568686 8021 raft_consensus.cc:374] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.568845 8021 raft_consensus.cc:729] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.569190 8021 consensus_queue.cc:260] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:54.570017 8021 ts_tablet_manager.cc:1398] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:12:54.569818 1668 catalog_manager.cc:5484] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:54.579739 8091 ts_tablet_manager.cc:1888] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:12:54.582123 4945 tablet_service.cc:1514] Processing DeleteTablet for tablet 84831077defe434c9f29eecfae7b44c8 with delete_type TABLET_DATA_TOMBSTONED (TS 00b61625d8a7460892c118a2570afda5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:54.582513 8084 tablet_replica.cc:330] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:54.582732 8084 raft_consensus.cc:2229] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.582954 8084 raft_consensus.cc:2258] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.584244 8084 ts_tablet_manager.cc:1875] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:54.592669 8089 ts_tablet_manager.cc:1888] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:54.592851 8089 log.cc:1196] T b48714749469474886f9e1d7e7635915 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/b48714749469474886f9e1d7e7635915
I20240503 08:12:54.594501 8084 ts_tablet_manager.cc:1888] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:54.594759 8084 log.cc:1196] T 84831077defe434c9f29eecfae7b44c8 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/84831077defe434c9f29eecfae7b44c8
I20240503 08:12:54.596858 8078 tablet_copy_client.cc:783] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:54.597313 8078 tablet_copy_client.cc:647] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:54.597929 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 84831077defe434c9f29eecfae7b44c8 (table rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]) successfully deleted
I20240503 08:12:54.606041 4698 consensus_queue.cc:237] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:12:54.608983 1943 raft_consensus.cc:1261] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:54.610317 7940 consensus_queue.cc:1035] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:54.611968 8093 tablet_bootstrap.cc:492] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:54.612286 8093 ts_tablet_manager.cc:1367] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.204s user 0.003s sys 0.004s
I20240503 08:12:54.614219 8091 log.cc:1196] T 54fe406b5f324d629243f7f939535620 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/54fe406b5f324d629243f7f939535620
I20240503 08:12:54.614513 7940 raft_consensus.cc:2932] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:54.615239 8093 raft_consensus.cc:348] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:54.615602 8093 raft_consensus.cc:374] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.615437 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 54fe406b5f324d629243f7f939535620 (table rebalancer_extra_table_8 [id=dde4f7cfffd44f36873e4e6dd5f3a917]) successfully deleted
I20240503 08:12:54.615783 8093 raft_consensus.cc:729] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.617534 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet b48714749469474886f9e1d7e7635915 (table rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]) successfully deleted
I20240503 08:12:54.617913 8093 consensus_queue.cc:260] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:54.619045 8093 ts_tablet_manager.cc:1398] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:54.623296 1943 raft_consensus.cc:2932] T 275fc5bbb1494306b1139b7f950ef80f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:54.626080 8078 tablet_copy_client.cc:523] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:54.628852 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 275fc5bbb1494306b1139b7f950ef80f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:54.630810 7941 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:54.635768 2201 raft_consensus.cc:1261] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:54.636054 7992 consensus_queue.cc:1035] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.636807 1813 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:54.639564 7179 consensus_queue.cc:1035] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:54.641698 2201 raft_consensus.cc:2932] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:54.644544 8078 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:54.657795 4963 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:54.651674 1669 catalog_manager.cc:5484] T 275fc5bbb1494306b1139b7f950ef80f P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:54.670006 1663 catalog_manager.cc:5484] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:12:54.677270 4812 tablet_service.cc:1514] Processing DeleteTablet for tablet 275fc5bbb1494306b1139b7f950ef80f with delete_type TABLET_DATA_TOMBSTONED (TS f24648c92c384f1ba349eee801f985e2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:54.677659 8075 tablet_replica.cc:330] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:54.677934 8075 raft_consensus.cc:2229] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:54.678216 8075 raft_consensus.cc:2258] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:54.679885 8075 ts_tablet_manager.cc:1875] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:54.687474 7992 consensus_queue.cc:1035] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:54.689505 8075 ts_tablet_manager.cc:1888] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:54.689903 8075 log.cc:1196] T 275fc5bbb1494306b1139b7f950ef80f P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/275fc5bbb1494306b1139b7f950ef80f
I20240503 08:12:54.690930 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet 275fc5bbb1494306b1139b7f950ef80f (table rebalancer_extra_table_26 [id=01490def2fa84e8ab1ccc8f8499d37cc]) successfully deleted
I20240503 08:12:54.805147 4967 consensus_queue.cc:237] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
W20240503 08:12:54.817495 4901 consensus_peers.cc:485] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a0bffb0aa304bd184b8ad26bc94fd18. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:54.818183 1813 raft_consensus.cc:1261] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:54.818894 8041 consensus_queue.cc:1035] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:54.824311 4698 raft_consensus.cc:1261] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:54.826901 8041 consensus_queue.cc:1035] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:54.843492 8130 ts_tablet_manager.cc:902] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:54.843860 8064 raft_consensus.cc:2932] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:54.848789 4698 raft_consensus.cc:2932] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:54.856024 1813 raft_consensus.cc:2932] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:54.860031 7992 raft_consensus.cc:482] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:54.860229 7992 raft_consensus.cc:504] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:54.861049 7992 leader_election.cc:288] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:54.861429 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da67bb92e824275b0e12f97ad95e102" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:54.861742 1813 raft_consensus.cc:2454] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:54.862203 4637 leader_election.cc:302] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:54.862464 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da67bb92e824275b0e12f97ad95e102" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:54.862761 4567 raft_consensus.cc:2454] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:54.864059 7992 raft_consensus.cc:2781] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:54.864187 7992 raft_consensus.cc:482] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:54.864285 7992 raft_consensus.cc:3037] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.865823 1669 catalog_manager.cc:5484] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:54.879524 7992 raft_consensus.cc:504] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:54.886492 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da67bb92e824275b0e12f97ad95e102" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:54.886636 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da67bb92e824275b0e12f97ad95e102" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:54.886790 4567 raft_consensus.cc:3037] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.886881 1813 raft_consensus.cc:3037] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:54.890983 1813 raft_consensus.cc:2454] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:54.892033 4637 leader_election.cc:302] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:54.892414 7940 raft_consensus.cc:2781] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:54.892859 7940 raft_consensus.cc:686] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:54.892764 1665 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:54.893141 7940 consensus_queue.cc:237] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:12:54.898636 4567 raft_consensus.cc:2454] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:54.899096 7992 leader_election.cc:288] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:54.906638 4812 tablet_service.cc:1467] Processing CreateTablet for tablet a03edb25e08e495c8dcb46adc9d8d2a4 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=88bd25fd205149ce87c2edef8849bd38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.907153 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03edb25e08e495c8dcb46adc9d8d2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.907074 2183 tablet_service.cc:1467] Processing CreateTablet for tablet a03edb25e08e495c8dcb46adc9d8d2a4 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=88bd25fd205149ce87c2edef8849bd38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.907558 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03edb25e08e495c8dcb46adc9d8d2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.907488 2054 tablet_service.cc:1467] Processing CreateTablet for tablet e613a6cbac2c4ad9b71442baa71ba9a3 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=88bd25fd205149ce87c2edef8849bd38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.907666 2052 tablet_service.cc:1467] Processing CreateTablet for tablet a03edb25e08e495c8dcb46adc9d8d2a4 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=88bd25fd205149ce87c2edef8849bd38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.907931 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e613a6cbac2c4ad9b71442baa71ba9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.908149 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03edb25e08e495c8dcb46adc9d8d2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.908149 4683 tablet_service.cc:1467] Processing CreateTablet for tablet e613a6cbac2c4ad9b71442baa71ba9a3 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=88bd25fd205149ce87c2edef8849bd38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.908528 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e613a6cbac2c4ad9b71442baa71ba9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.908607 4814 tablet_service.cc:1467] Processing CreateTablet for tablet e613a6cbac2c4ad9b71442baa71ba9a3 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=88bd25fd205149ce87c2edef8849bd38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.909018 4814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e613a6cbac2c4ad9b71442baa71ba9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.924422 1924 tablet_service.cc:1467] Processing CreateTablet for tablet a97625d919124c559b1e9a5cb785153f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=88bd25fd205149ce87c2edef8849bd38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.924922 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a97625d919124c559b1e9a5cb785153f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.927807 4813 tablet_service.cc:1467] Processing CreateTablet for tablet a97625d919124c559b1e9a5cb785153f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=88bd25fd205149ce87c2edef8849bd38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.928242 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a97625d919124c559b1e9a5cb785153f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.929410 2184 tablet_service.cc:1467] Processing CreateTablet for tablet a97625d919124c559b1e9a5cb785153f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=88bd25fd205149ce87c2edef8849bd38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:54.929843 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a97625d919124c559b1e9a5cb785153f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:54.933385 8078 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:54.933827 8078 tablet_bootstrap.cc:492] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:12:54.934144 8078 ts_tablet_manager.cc:1367] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.291s user 0.052s sys 0.008s
I20240503 08:12:54.935068 8078 raft_consensus.cc:348] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:54.937309 8136 tablet_bootstrap.cc:492] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:54.939574 8078 raft_consensus.cc:729] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:12:54.939911 8078 consensus_queue.cc:260] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:12:54.940765 8136 tablet_bootstrap.cc:654] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.941337 8078 ts_tablet_manager.cc:1398] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240503 08:12:54.944801 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-53197deeb8d7439c90b019a6a90857e0 received from {username='slave'} at 127.27.157.197:32893
I20240503 08:12:54.944954 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-53197deeb8d7439c90b019a6a90857e0 on tablet 53197deeb8d7439c90b019a6a90857e0 with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:12:54.945787 8093 tablet_bootstrap.cc:492] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:54.948102 8093 tablet_bootstrap.cc:654] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.986546 1669 catalog_manager.cc:5484] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:54.987836 8093 tablet_bootstrap.cc:492] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:54.988090 8093 ts_tablet_manager.cc:1367] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.002s
I20240503 08:12:54.988961 8139 tablet_bootstrap.cc:492] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:54.989213 8093 raft_consensus.cc:348] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:54.984817 8135 tablet_bootstrap.cc:492] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:54.990319 8093 raft_consensus.cc:374] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:54.991016 8093 raft_consensus.cc:729] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:54.992017 8093 consensus_queue.cc:260] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:54.998976 8093 ts_tablet_manager.cc:1398] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 08:12:55.003273 8135 tablet_bootstrap.cc:654] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:54.996366 8133 tablet_bootstrap.cc:492] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:55.009523 8139 tablet_bootstrap.cc:654] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.012670 8135 tablet_bootstrap.cc:492] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:55.013444 8135 ts_tablet_manager.cc:1367] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.003s
I20240503 08:12:55.015499 8135 raft_consensus.cc:348] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.016352 8135 raft_consensus.cc:374] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.016578 8135 raft_consensus.cc:729] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.017978 8135 consensus_queue.cc:260] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.020974 8135 ts_tablet_manager.cc:1398] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:55.022666 8135 tablet_bootstrap.cc:492] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:55.027770 8135 tablet_bootstrap.cc:654] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.033085 8135 tablet_bootstrap.cc:492] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:55.034273 8135 ts_tablet_manager.cc:1367] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 08:12:55.037451 8133 tablet_bootstrap.cc:654] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.040362 8135 raft_consensus.cc:348] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:55.040534 8136 tablet_bootstrap.cc:492] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:55.040823 8135 raft_consensus.cc:374] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.040737 8136 ts_tablet_manager.cc:1367] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.104s user 0.039s sys 0.026s
I20240503 08:12:55.042007 8135 raft_consensus.cc:729] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.042527 7940 raft_consensus.cc:482] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:55.042697 7940 raft_consensus.cc:504] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:55.042948 8135 consensus_queue.cc:260] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:55.043027 8136 raft_consensus.cc:348] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.043363 8136 raft_consensus.cc:374] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.043485 8136 raft_consensus.cc:729] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.043534 7940 leader_election.cc:288] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:55.044173 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e613a6cbac2c4ad9b71442baa71ba9a3" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:55.044165 8136 consensus_queue.cc:260] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.044397 2072 raft_consensus.cc:2454] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:55.045257 4636 leader_election.cc:302] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:55.045814 7940 raft_consensus.cc:2781] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:55.045930 7940 raft_consensus.cc:482] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:55.046010 7940 raft_consensus.cc:3037] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.047006 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e613a6cbac2c4ad9b71442baa71ba9a3" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:55.047067 8136 ts_tablet_manager.cc:1398] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:55.047526 8136 tablet_bootstrap.cc:492] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
W20240503 08:12:55.047940 4638 leader_election.cc:341] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:55.053889 8136 tablet_bootstrap.cc:654] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.058239 7940 raft_consensus.cc:504] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:55.059389 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e613a6cbac2c4ad9b71442baa71ba9a3" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:55.059633 2072 raft_consensus.cc:3037] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.062476 2072 raft_consensus.cc:2454] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
I20240503 08:12:55.063069 4636 leader_election.cc:302] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:55.063621 7992 raft_consensus.cc:2781] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:55.063766 7992 raft_consensus.cc:686] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:55.064047 7992 consensus_queue.cc:237] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:55.065771 7940 leader_election.cc:288] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:55.067500 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e613a6cbac2c4ad9b71442baa71ba9a3" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
W20240503 08:12:55.069382 4638 leader_election.cc:341] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:55.070941 8135 ts_tablet_manager.cc:1398] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240503 08:12:55.083088 8133 tablet_bootstrap.cc:492] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:55.083269 8133 ts_tablet_manager.cc:1367] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.087s user 0.002s sys 0.004s
I20240503 08:12:55.084240 8133 raft_consensus.cc:348] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.084491 8133 raft_consensus.cc:374] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.084560 8133 raft_consensus.cc:729] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.082823 1665 catalog_manager.cc:5484] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:55.084793 8133 consensus_queue.cc:260] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.085475 8133 ts_tablet_manager.cc:1398] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:55.092664 8133 tablet_bootstrap.cc:492] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:55.094830 4698 consensus_queue.cc:237] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:55.096354 8133 tablet_bootstrap.cc:654] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.100800 8065 raft_consensus.cc:482] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:55.101069 8065 raft_consensus.cc:504] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.102137 8065 leader_election.cc:288] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:55.103369 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03edb25e08e495c8dcb46adc9d8d2a4" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:55.103456 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03edb25e08e495c8dcb46adc9d8d2a4" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:55.103616 2204 raft_consensus.cc:2454] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:55.103668 2072 raft_consensus.cc:2454] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:55.104471 4768 leader_election.cc:302] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:55.105345 8065 raft_consensus.cc:2781] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:55.105453 8065 raft_consensus.cc:482] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:55.105558 8065 raft_consensus.cc:3037] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.107270 8139 tablet_bootstrap.cc:492] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:55.107537 8139 ts_tablet_manager.cc:1367] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.119s user 0.013s sys 0.027s
I20240503 08:12:55.109825 8065 raft_consensus.cc:504] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.110538 8065 leader_election.cc:288] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:55.111214 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03edb25e08e495c8dcb46adc9d8d2a4" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:55.111424 2204 raft_consensus.cc:3037] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:12:55.111559 4636 consensus_peers.cc:485] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0caddf5e39f8440baa8ad3656ce91a8e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:55.112933 4967 raft_consensus.cc:1261] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:55.113741 7992 consensus_queue.cc:1035] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:55.114261 2204 raft_consensus.cc:2454] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:55.117761 8133 tablet_bootstrap.cc:492] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:55.118000 8133 ts_tablet_manager.cc:1367] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240503 08:12:55.118618 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a03edb25e08e495c8dcb46adc9d8d2a4" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:55.118836 2072 raft_consensus.cc:3037] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.118809 8133 raft_consensus.cc:348] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.118992 8133 raft_consensus.cc:374] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.119050 8133 raft_consensus.cc:729] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.119290 8133 consensus_queue.cc:260] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.121670 8133 ts_tablet_manager.cc:1398] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 08:12:55.122073 1813 raft_consensus.cc:1261] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:55.122732 7992 consensus_queue.cc:1035] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:55.125594 8111 raft_consensus.cc:2932] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:55.126251 4963 raft_consensus.cc:2932] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:55.129801 4768 leader_election.cc:302] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:55.134048 8065 raft_consensus.cc:2781] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:55.134519 8065 raft_consensus.cc:686] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:55.135282 8065 consensus_queue.cc:237] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.135079 1669 catalog_manager.cc:5484] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:55.138155 8130 tablet_copy_client.cc:308] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:41181
I20240503 08:12:55.141027 1833 tablet_copy_service.cc:140] P b8f581390da449ccb9e299011296405e: Received BeginTabletCopySession request for tablet 3219159f92774adbb7ecaa77ac267021 from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:57509)
I20240503 08:12:55.141194 1833 tablet_copy_service.cc:161] P b8f581390da449ccb9e299011296405e: Beginning new tablet copy session on tablet 3219159f92774adbb7ecaa77ac267021 from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:57509: session id = 00b61625d8a7460892c118a2570afda5-3219159f92774adbb7ecaa77ac267021
I20240503 08:12:55.151168 8136 tablet_bootstrap.cc:492] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:55.151549 8136 ts_tablet_manager.cc:1367] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.104s user 0.009s sys 0.016s
I20240503 08:12:55.151549 2072 raft_consensus.cc:2454] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:55.152807 8136 raft_consensus.cc:348] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:55.153108 8136 raft_consensus.cc:374] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.153219 8136 raft_consensus.cc:729] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.153604 8136 consensus_queue.cc:260] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:55.154196 8136 ts_tablet_manager.cc:1398] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:55.154675 8136 tablet_bootstrap.cc:492] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:55.164058 1833 tablet_copy_source_session.cc:205] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:55.166873 1812 raft_consensus.cc:2932] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:55.168318 8130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3219159f92774adbb7ecaa77ac267021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.180245 8136 tablet_bootstrap.cc:654] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.125087 7941 raft_consensus.cc:1050] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:12:55.190565 7941 consensus_queue.cc:237] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:55.193296 8136 tablet_bootstrap.cc:492] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:55.193497 8136 ts_tablet_manager.cc:1367] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.007s
I20240503 08:12:55.194766 8136 raft_consensus.cc:348] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.195039 8136 raft_consensus.cc:374] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.195181 8136 raft_consensus.cc:729] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.197625 8136 consensus_queue.cc:260] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.201686 8136 ts_tablet_manager.cc:1398] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 08:12:55.212502 8130 tablet_copy_client.cc:783] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:55.213021 8130 tablet_copy_client.cc:647] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:55.215690 2204 raft_consensus.cc:1261] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:55.217700 7964 consensus_queue.cc:1035] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:55.218086 8130 tablet_copy_client.cc:523] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:55.219661 4698 raft_consensus.cc:1261] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:55.220512 8117 consensus_queue.cc:1035] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:55.221768 1669 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:55.230317 8130 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:55.235340 4567 raft_consensus.cc:1261] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:55.237277 7941 consensus_queue.cc:1035] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:55.254206 8117 raft_consensus.cc:2932] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:55.255040 2204 raft_consensus.cc:2932] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:55.267855 1669 catalog_manager.cc:5484] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:55.108434 8139 raft_consensus.cc:348] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.282291 8139 raft_consensus.cc:374] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:55.284544 8139 raft_consensus.cc:729] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:55.284498 4833 consensus_queue.cc:237] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:55.281174 4698 raft_consensus.cc:2932] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:55.285025 8139 consensus_queue.cc:260] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.285840 8139 ts_tablet_manager.cc:1398] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.178s user 0.000s sys 0.002s
I20240503 08:12:55.293284 4566 raft_consensus.cc:1261] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:12:55.293949 4769 consensus_peers.cc:485] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e641fe703c14f4cb1bd22c676d79576. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:55.294270 7742 consensus_queue.cc:1035] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:55.297493 8162 ts_tablet_manager.cc:902] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:55.300154 1813 raft_consensus.cc:1261] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:55.300451 4568 raft_consensus.cc:1261] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:55.301157 8111 consensus_queue.cc:1035] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.301930 8161 consensus_queue.cc:1035] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240503 08:12:55.304658 4768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.200:45283, user_credentials={real_user=slave}} blocked reactor thread for 147514us
I20240503 08:12:55.312920 4569 raft_consensus.cc:2932] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:55.314069 7742 raft_consensus.cc:2932] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:55.319852 1813 raft_consensus.cc:2932] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:55.324204 1812 raft_consensus.cc:1261] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:55.324868 1941 consensus_queue.cc:237] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:55.325160 8111 consensus_queue.cc:1035] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.329795 1665 catalog_manager.cc:5576] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465) does not have the latest schema for tablet 9da67bb92e824275b0e12f97ad95e102 (table rebalancer_dynamic_table [id=b315b0aaa3124d5081e0d226641eb2da]). Expected version 1 got 0
I20240503 08:12:55.332556 4568 raft_consensus.cc:2932] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:55.339944 1667 catalog_manager.cc:5484] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:55.341650 2204 raft_consensus.cc:1261] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:55.342615 4566 raft_consensus.cc:1261] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:55.343216 7964 consensus_queue.cc:1035] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:55.343894 7964 consensus_queue.cc:1035] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:55.349503 7964 raft_consensus.cc:2932] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:55.351882 4569 raft_consensus.cc:2932] T 47dde43816964e89bfb235aad34e9583 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:55.355310 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 47dde43816964e89bfb235aad34e9583 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:55.357789 2201 raft_consensus.cc:2932] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:55.369302 1663 catalog_manager.cc:5484] T 47dde43816964e89bfb235aad34e9583 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:55.374748 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 47dde43816964e89bfb235aad34e9583 with delete_type TABLET_DATA_TOMBSTONED (TS 31f41542452b4b1eb994ff4223d039fc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:55.357374 1665 catalog_manager.cc:5484] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:55.401504 8183 tablet.cc:1722] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:55.402535 8183 tablet.cc:1620] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:55.405306 1663 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 9da67bb92e824275b0e12f97ad95e102 (table rebalancer_dynamic_table [id=b315b0aaa3124d5081e0d226641eb2da]). Expected version 1 got 0
I20240503 08:12:55.408061 8193 tablet.cc:1722] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:55.409512 8174 raft_consensus.cc:482] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:55.409927 8174 raft_consensus.cc:504] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.411088 2204 consensus_queue.cc:237] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:55.411944 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a97625d919124c559b1e9a5cb785153f" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:55.412146 2201 raft_consensus.cc:2454] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:55.412295 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a97625d919124c559b1e9a5cb785153f" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:55.412535 1878 leader_election.cc:302] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:55.412567 4834 raft_consensus.cc:2454] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:55.413126 8117 raft_consensus.cc:2781] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:55.413278 8117 raft_consensus.cc:482] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:55.413378 8117 raft_consensus.cc:3037] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.416965 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=b315b0aaa3124d5081e0d226641eb2da] alter complete (version 1)
I20240503 08:12:55.418109 8117 raft_consensus.cc:504] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.420253 8117 leader_election.cc:288] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:55.420729 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a97625d919124c559b1e9a5cb785153f" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:55.421113 4834 raft_consensus.cc:3037] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.421871 8193 tablet.cc:1620] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:55.422470 8174 leader_election.cc:288] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:55.422812 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a97625d919124c559b1e9a5cb785153f" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:55.423000 2201 raft_consensus.cc:3037] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:12:55.423125 2140 consensus_peers.cc:485] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df06752e8d7247dd962a376c629473c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:55.423902 4963 raft_consensus.cc:1261] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:55.425280 7924 consensus_queue.cc:1035] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:55.425791 2201 raft_consensus.cc:2454] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:55.429921 1878 leader_election.cc:302] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:55.431352 4569 raft_consensus.cc:1261] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:55.432258 7964 raft_consensus.cc:2781] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:55.432155 7924 consensus_queue.cc:1035] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:55.432404 7964 raft_consensus.cc:686] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:55.432722 7964 consensus_queue.cc:237] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:12:55.448112 4834 raft_consensus.cc:2454] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:55.449107 8192 tablet_replica.cc:330] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:55.449435 8192 raft_consensus.cc:2229] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:55.450439 8192 raft_consensus.cc:2258] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:55.451719 8192 ts_tablet_manager.cc:1875] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:55.452297 4569 raft_consensus.cc:1203] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:12:55.454810 1669 catalog_manager.cc:5576] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181) does not have the latest schema for tablet 9da67bb92e824275b0e12f97ad95e102 (table rebalancer_dynamic_table [id=b315b0aaa3124d5081e0d226641eb2da]). Expected version 1 got 0
I20240503 08:12:55.461501 8201 ts_tablet_manager.cc:902] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:55.452415 8065 raft_consensus.cc:482] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:55.462286 8065 raft_consensus.cc:504] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:55.463394 8065 leader_election.cc:288] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:55.467944 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e613a6cbac2c4ad9b71442baa71ba9a3" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:55.472844 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e613a6cbac2c4ad9b71442baa71ba9a3" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:55.472851 8197 tablet.cc:1722] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:55.473162 2072 raft_consensus.cc:2379] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f24648c92c384f1ba349eee801f985e2 in current term 1: Already voted for candidate 31f41542452b4b1eb994ff4223d039fc in this term.
I20240503 08:12:55.473397 8197 tablet.cc:1620] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:55.474195 4768 leader_election.cc:302] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f24648c92c384f1ba349eee801f985e2; no voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:12:55.475126 7742 raft_consensus.cc:3037] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:55.477965 7924 raft_consensus.cc:2932] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:55.479954 7742 raft_consensus.cc:2726] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 08:12:55.454768 4767 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.197:42465, user_credentials={real_user=slave}} blocked reactor thread for 93237.3us
I20240503 08:12:55.483518 4963 raft_consensus.cc:2932] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:55.489914 1668 catalog_manager.cc:5484] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:55.491197 1667 catalog_manager.cc:5576] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181) does not have the latest schema for tablet 9da67bb92e824275b0e12f97ad95e102 (table rebalancer_dynamic_table [id=b315b0aaa3124d5081e0d226641eb2da]). Expected version 1 got 0
I20240503 08:12:55.493309 4568 raft_consensus.cc:2932] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:12:55.512749 1669 catalog_manager.cc:5484] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:55.513509 8192 ts_tablet_manager.cc:1888] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:55.514268 8192 log.cc:1196] T 47dde43816964e89bfb235aad34e9583 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/47dde43816964e89bfb235aad34e9583
I20240503 08:12:55.515461 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 47dde43816964e89bfb235aad34e9583 (table rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]) successfully deleted
I20240503 08:12:55.516125 8182 consensus_queue.cc:1035] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.525125 1668 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:55.539166 8183 tablet.cc:1722] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:55.539654 8111 consensus_queue.cc:1035] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.539853 8183 tablet.cc:1620] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:55.546402 8193 tablet.cc:1722] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:55.547022 8193 tablet.cc:1620] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: MemRowSet was empty: no flush needed.
I20240503 08:12:55.552197 8162 tablet_copy_client.cc:308] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:12:55.553346 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:51509)
I20240503 08:12:55.553557 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:51509: session id = f24648c92c384f1ba349eee801f985e2-1cf9b2eef42e4fff8cdf5f9dd7b8b31e
I20240503 08:12:55.556726 1964 tablet_copy_source_session.cc:205] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:55.561944 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=b315b0aaa3124d5081e0d226641eb2da] alter complete (version 2)
I20240503 08:12:55.570791 8162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.601305 8197 tablet.cc:1722] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:55.602895 1668 catalog_manager.cc:5576] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181) does not have the latest schema for tablet 9da67bb92e824275b0e12f97ad95e102 (table rebalancer_dynamic_table [id=b315b0aaa3124d5081e0d226641eb2da]). Expected version 2 got 1
I20240503 08:12:55.603922 8197 tablet.cc:1620] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: MemRowSet was empty: no flush needed.
I20240503 08:12:55.607805 8130 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:55.608506 8130 tablet_bootstrap.cc:492] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:12:55.608820 8130 ts_tablet_manager.cc:1367] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.379s user 0.042s sys 0.022s
I20240503 08:12:55.610697 8130 raft_consensus.cc:348] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:55.610998 8130 raft_consensus.cc:729] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:12:55.611204 8130 consensus_queue.cc:260] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:12:55.614437 8130 ts_tablet_manager.cc:1398] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 08:12:55.620055 1833 tablet_copy_service.cc:342] P b8f581390da449ccb9e299011296405e: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-3219159f92774adbb7ecaa77ac267021 received from {username='slave'} at 127.27.157.200:57509
I20240503 08:12:55.620201 1833 tablet_copy_service.cc:433] P b8f581390da449ccb9e299011296405e: ending tablet copy session 00b61625d8a7460892c118a2570afda5-3219159f92774adbb7ecaa77ac267021 on tablet 3219159f92774adbb7ecaa77ac267021 with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:12:55.641860 8162 tablet_copy_client.cc:783] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:55.650327 8162 tablet_copy_client.cc:647] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:55.653717 8162 tablet_copy_client.cc:523] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:55.659274 8065 consensus_queue.cc:1035] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.661820 8162 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:55.663429 4967 consensus_queue.cc:237] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:55.665642 1668 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:55.680480 4566 raft_consensus.cc:1261] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:55.681520 8183 tablet.cc:1722] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:55.682999 4834 raft_consensus.cc:1261] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:55.683910 8064 consensus_queue.cc:1035] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:55.684612 8064 consensus_queue.cc:1035] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:55.684649 8193 tablet.cc:1722] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 08:12:55.693852 4901 consensus_peers.cc:485] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09a268c672ba4580a2d93b5d15d12666. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:55.703285 8197 tablet.cc:1722] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:55.704016 1665 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=b315b0aaa3124d5081e0d226641eb2da] alter complete (version 3)
I20240503 08:12:55.711927 8221 ts_tablet_manager.cc:902] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:55.712711 8041 raft_consensus.cc:2932] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:55.714942 7742 consensus_queue.cc:1035] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.722959 8201 tablet_copy_client.cc:308] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:12:55.722591 4568 raft_consensus.cc:2932] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:55.723773 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet 9a0bffb0aa304bd184b8ad26bc94fd18 from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:40265)
I20240503 08:12:55.723985 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet 9a0bffb0aa304bd184b8ad26bc94fd18 from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:40265: session id = 184c543cb05d49cc98ba1c56617be7a5-9a0bffb0aa304bd184b8ad26bc94fd18
I20240503 08:12:55.727110 4988 tablet_copy_source_session.cc:205] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:55.729665 8201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0bffb0aa304bd184b8ad26bc94fd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.735169 4967 raft_consensus.cc:1203] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:12:55.740206 8225 ts_tablet_manager.cc:902] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a: Initiating tablet copy from peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:55.746812 1669 catalog_manager.cc:5484] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:55.769503 8201 tablet_copy_client.cc:783] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:55.769924 8201 tablet_copy_client.cc:647] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:55.785924 8201 tablet_copy_client.cc:523] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:55.792948 8201 tablet_bootstrap.cc:492] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:55.793587 8221 tablet_copy_client.cc:308] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:12:55.795428 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet 1e641fe703c14f4cb1bd22c676d79576 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a ({username='slave'} at 127.27.157.194:55239)
I20240503 08:12:55.795689 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet 1e641fe703c14f4cb1bd22c676d79576 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a at {username='slave'} at 127.27.157.194:55239: session id = 315f3f0bcbf447a4b93b1aafcfbf0f5a-1e641fe703c14f4cb1bd22c676d79576
I20240503 08:12:55.800527 8225 tablet_copy_client.cc:308] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.198:46559
I20240503 08:12:55.809736 4855 tablet_copy_source_session.cc:205] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:55.811683 1669 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:55.812047 1669 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:55.812227 8221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e641fe703c14f4cb1bd22c676d79576. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.814389 4723 tablet_copy_service.cc:140] P 31f41542452b4b1eb994ff4223d039fc: Received BeginTabletCopySession request for tablet 0caddf5e39f8440baa8ad3656ce91a8e from peer 43e2e01cdd314f9c872c4336ef74250a ({username='slave'} at 127.27.157.195:37471)
I20240503 08:12:55.814667 4723 tablet_copy_service.cc:161] P 31f41542452b4b1eb994ff4223d039fc: Beginning new tablet copy session on tablet 0caddf5e39f8440baa8ad3656ce91a8e from peer 43e2e01cdd314f9c872c4336ef74250a at {username='slave'} at 127.27.157.195:37471: session id = 43e2e01cdd314f9c872c4336ef74250a-0caddf5e39f8440baa8ad3656ce91a8e
I20240503 08:12:55.817034 1669 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 9da67bb92e824275b0e12f97ad95e102
I20240503 08:12:55.817860 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da67bb92e824275b0e12f97ad95e102 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:55 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:55.818176 8192 tablet_replica.cc:330] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:55.818202 4723 tablet_copy_source_session.cc:205] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:55.818408 8192 raft_consensus.cc:2229] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:55.818791 8192 raft_consensus.cc:2258] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:55.820478 4547 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da67bb92e824275b0e12f97ad95e102 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:55 UTC) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:55.821563 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da67bb92e824275b0e12f97ad95e102 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:55 UTC) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:55.824918 8192 ts_tablet_manager.cc:1875] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:55.825712 8221 tablet_copy_client.cc:783] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:55.826001 8221 tablet_copy_client.cc:647] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:55.837815 8225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0caddf5e39f8440baa8ad3656ce91a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.849323 8201 tablet_bootstrap.cc:492] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:55.849753 8201 tablet_bootstrap.cc:492] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:12:55.850025 8201 ts_tablet_manager.cc:1367] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.006s
I20240503 08:12:55.852316 8225 tablet_copy_client.cc:783] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:55.852636 8225 tablet_copy_client.cc:647] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:55.855262 8225 tablet_copy_client.cc:523] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:55.861172 8221 tablet_copy_client.cc:523] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:55.867311 8117 consensus_queue.cc:1035] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.854672 8201 raft_consensus.cc:348] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:55.873625 8201 raft_consensus.cc:729] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:12:55.874570 8201 consensus_queue.cc:260] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:55.875588 8221 tablet_bootstrap.cc:492] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:55.881320 8201 ts_tablet_manager.cc:1398] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.031s user 0.004s sys 0.001s
I20240503 08:12:55.884601 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-9a0bffb0aa304bd184b8ad26bc94fd18 received from {username='slave'} at 127.27.157.196:40265
I20240503 08:12:55.884804 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-9a0bffb0aa304bd184b8ad26bc94fd18 on tablet 9a0bffb0aa304bd184b8ad26bc94fd18 with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:55.893471 8192 ts_tablet_manager.cc:1888] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:55.893529 8225 tablet_bootstrap.cc:492] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:55.917987 8238 tablet_replica.cc:330] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:55.918627 8238 raft_consensus.cc:2229] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:55.918998 8238 raft_consensus.cc:2258] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:55.920305 8238 ts_tablet_manager.cc:1875] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:55.922933 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:55.924273 8239 tablet_replica.cc:330] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:55.924643 8239 raft_consensus.cc:2229] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:55.924979 8239 raft_consensus.cc:2258] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:55.928107 8221 tablet_bootstrap.cc:492] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:55.928573 8221 tablet_bootstrap.cc:492] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap complete.
I20240503 08:12:55.928730 8117 consensus_queue.cc:1035] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:55.928856 8221 ts_tablet_manager.cc:1367] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.000s
I20240503 08:12:55.929929 8221 raft_consensus.cc:348] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:55.931243 8221 raft_consensus.cc:729] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: LEARNER
I20240503 08:12:55.931488 8221 consensus_queue.cc:260] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:55.933701 8221 ts_tablet_manager.cc:1398] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:12:55.936285 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-1e641fe703c14f4cb1bd22c676d79576 received from {username='slave'} at 127.27.157.194:55239
I20240503 08:12:55.936511 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-1e641fe703c14f4cb1bd22c676d79576 on tablet 1e641fe703c14f4cb1bd22c676d79576 with peer 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:12:55.938045 8192 log.cc:1196] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/9da67bb92e824275b0e12f97ad95e102
I20240503 08:12:55.939680 8192 ts_tablet_manager.cc:1909] T 9da67bb92e824275b0e12f97ad95e102 P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:55.950503 8239 ts_tablet_manager.cc:1875] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:55.963302 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=163bce7cc559485da3b201768dbcf989]), partition=
I20240503 08:12:55.963932 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.969033 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=163bce7cc559485da3b201768dbcf989]), partition=
I20240503 08:12:55.969440 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:55.984797 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 9da67bb92e824275b0e12f97ad95e102 (table rebalancer_dynamic_table_new_name [id=b315b0aaa3124d5081e0d226641eb2da]) successfully deleted
I20240503 08:12:55.985947 8174 raft_consensus.cc:1050] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:12:55.987103 8174 consensus_queue.cc:237] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:55.992198 8254 tablet_bootstrap.cc:492] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:55.996798 8254 tablet_bootstrap.cc:654] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:55.997273 8243 ts_tablet_manager.cc:902] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
W20240503 08:12:56.007519 4903 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.199:39455, user_credentials={real_user=slave}} blocked reactor thread for 106425us
I20240503 08:12:56.011262 4834 raft_consensus.cc:2932] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
W20240503 08:12:56.013069 4902 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.193:41181, user_credentials={real_user=slave}} blocked reactor thread for 112090us
I20240503 08:12:56.019299 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=163bce7cc559485da3b201768dbcf989]), partition=
I20240503 08:12:56.019711 8238 ts_tablet_manager.cc:1888] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:56.019778 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.019868 8238 log.cc:1196] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/9da67bb92e824275b0e12f97ad95e102
I20240503 08:12:56.020395 8238 ts_tablet_manager.cc:1909] T 9da67bb92e824275b0e12f97ad95e102 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting consensus metadata
I20240503 08:12:56.023756 8239 ts_tablet_manager.cc:1888] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:56.023898 8239 log.cc:1196] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/9da67bb92e824275b0e12f97ad95e102
I20240503 08:12:56.024314 8239 ts_tablet_manager.cc:1909] T 9da67bb92e824275b0e12f97ad95e102 P b8f581390da449ccb9e299011296405e: Deleting consensus metadata
I20240503 08:12:56.025588 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 9da67bb92e824275b0e12f97ad95e102 (table rebalancer_dynamic_table_new_name [id=b315b0aaa3124d5081e0d226641eb2da]) successfully deleted
I20240503 08:12:56.029021 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 9da67bb92e824275b0e12f97ad95e102 (table rebalancer_dynamic_table_new_name [id=b315b0aaa3124d5081e0d226641eb2da]) successfully deleted
I20240503 08:12:56.031049 8261 tablet_bootstrap.cc:492] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:56.042928 8260 tablet_bootstrap.cc:492] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:56.042448 8261 tablet_bootstrap.cc:654] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.045979 8260 tablet_bootstrap.cc:654] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.052454 2204 raft_consensus.cc:1203] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:56.053459 4566 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:12:56.054056 7941 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:56.065282 1812 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:12:56.066140 7941 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:56.068593 8261 tablet_bootstrap.cc:492] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:56.069324 4835 consensus_queue.cc:237] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:56.073527 8261 ts_tablet_manager.cc:1367] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.000s
I20240503 08:12:56.078823 8261 raft_consensus.cc:348] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.080670 8261 raft_consensus.cc:374] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.082013 8261 raft_consensus.cc:729] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.083222 8261 consensus_queue.cc:260] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.085578 8261 ts_tablet_manager.cc:1398] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240503 08:12:56.088382 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 08:12:56.090807 4769 consensus_peers.cc:485] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 929d8d354b914502b5e677ccc84e2eda. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:56.093147 2072 raft_consensus.cc:1261] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:56.094061 8065 consensus_queue.cc:1035] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:56.110677 8065 raft_consensus.cc:2932] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:56.111369 4547 tablet_service.cc:1467] Processing CreateTablet for tablet 438146d5fa194156b929d7190b11a5de (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=734b33fe6b21405a806dd0d9942ffd95]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.111876 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438146d5fa194156b929d7190b11a5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.115418 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 12b0c4a0b0564a279806d3fd6f981eed (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=734b33fe6b21405a806dd0d9942ffd95]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.115743 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 438146d5fa194156b929d7190b11a5de (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=734b33fe6b21405a806dd0d9942ffd95]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.116039 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b0c4a0b0564a279806d3fd6f981eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.116216 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438146d5fa194156b929d7190b11a5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.121109 1793 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3070c262ed420a91ac0ed063d16471 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=734b33fe6b21405a806dd0d9942ffd95]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.121587 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3070c262ed420a91ac0ed063d16471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.121629 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 12b0c4a0b0564a279806d3fd6f981eed (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=734b33fe6b21405a806dd0d9942ffd95]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.122174 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b0c4a0b0564a279806d3fd6f981eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.122905 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 12b0c4a0b0564a279806d3fd6f981eed (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=734b33fe6b21405a806dd0d9942ffd95]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.123867 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b0c4a0b0564a279806d3fd6f981eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.126976 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3070c262ed420a91ac0ed063d16471 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=734b33fe6b21405a806dd0d9942ffd95]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.127547 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3070c262ed420a91ac0ed063d16471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.131847 2073 raft_consensus.cc:2932] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:56.152519 8272 tablet_bootstrap.cc:492] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:56.154305 8272 tablet_bootstrap.cc:654] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.156419 1665 catalog_manager.cc:5484] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:56.181164 8271 tablet_bootstrap.cc:492] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:56.184600 8271 tablet_bootstrap.cc:654] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.194587 4698 raft_consensus.cc:1261] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:56.208577 8271 tablet_bootstrap.cc:492] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:56.208875 8271 ts_tablet_manager.cc:1367] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.003s
I20240503 08:12:56.209653 8275 tablet_bootstrap.cc:492] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:56.209918 8271 raft_consensus.cc:348] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.210299 8271 raft_consensus.cc:374] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.210451 8271 raft_consensus.cc:729] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.210829 8271 consensus_queue.cc:260] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.217741 8272 tablet_bootstrap.cc:492] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:56.217980 8272 ts_tablet_manager.cc:1367] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.003s
I20240503 08:12:56.218704 8272 raft_consensus.cc:348] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:56.218988 8272 raft_consensus.cc:374] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.219095 8272 raft_consensus.cc:729] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.219416 8272 consensus_queue.cc:260] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:56.214318 8275 tablet_bootstrap.cc:654] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.223570 7737 raft_consensus.cc:482] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:56.223750 7737 raft_consensus.cc:504] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:56.224579 7737 leader_election.cc:288] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:56.226003 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3070c262ed420a91ac0ed063d16471" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
W20240503 08:12:56.226627 1747 leader_election.cc:341] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Illegal state: must be running to vote when last-logged opid is not known
W20240503 08:12:56.230924 1749 leader_election.cc:341] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Not found: Tablet not found: 0c3070c262ed420a91ac0ed063d16471
I20240503 08:12:56.231093 1749 leader_election.cc:302] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f581390da449ccb9e299011296405e; no voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:12:56.231550 7737 raft_consensus.cc:2726] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:56.235910 8275 tablet_bootstrap.cc:492] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:56.236202 8275 ts_tablet_manager.cc:1367] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20240503 08:12:56.237113 8275 raft_consensus.cc:348] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.237426 8275 raft_consensus.cc:374] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.237548 8275 raft_consensus.cc:729] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.237879 8275 consensus_queue.cc:260] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.240520 8275 ts_tablet_manager.cc:1398] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:56.240916 8275 tablet_bootstrap.cc:492] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:56.242651 4967 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:12:56.207463 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3070c262ed420a91ac0ed063d16471 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=734b33fe6b21405a806dd0d9942ffd95]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.244215 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3070c262ed420a91ac0ed063d16471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.244709 8065 consensus_queue.cc:1035] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:56.255362 8201 ts_tablet_manager.cc:902] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:56.259210 7964 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:56.266228 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 438146d5fa194156b929d7190b11a5de (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=734b33fe6b21405a806dd0d9942ffd95]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:56.266820 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438146d5fa194156b929d7190b11a5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.267375 8272 ts_tablet_manager.cc:1398] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.049s user 0.001s sys 0.000s
I20240503 08:12:56.267712 8272 tablet_bootstrap.cc:492] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:56.270598 8272 tablet_bootstrap.cc:654] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.276928 8261 tablet_bootstrap.cc:492] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:56.248907 8225 tablet_bootstrap.cc:492] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:56.278107 8225 tablet_bootstrap.cc:492] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap complete.
I20240503 08:12:56.278416 8225 ts_tablet_manager.cc:1367] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.385s user 0.041s sys 0.061s
I20240503 08:12:56.279091 8261 tablet_bootstrap.cc:654] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.279160 8225 raft_consensus.cc:348] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:56.279413 8225 raft_consensus.cc:729] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: LEARNER
I20240503 08:12:56.279615 8225 consensus_queue.cc:260] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:56.280158 8225 ts_tablet_manager.cc:1398] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:12:56.280966 4723 tablet_copy_service.cc:342] P 31f41542452b4b1eb994ff4223d039fc: Request end of tablet copy session 43e2e01cdd314f9c872c4336ef74250a-0caddf5e39f8440baa8ad3656ce91a8e received from {username='slave'} at 127.27.157.195:37471
I20240503 08:12:56.281152 4723 tablet_copy_service.cc:433] P 31f41542452b4b1eb994ff4223d039fc: ending tablet copy session 43e2e01cdd314f9c872c4336ef74250a-0caddf5e39f8440baa8ad3656ce91a8e on tablet 0caddf5e39f8440baa8ad3656ce91a8e with peer 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:12:56.242755 8275 tablet_bootstrap.cc:654] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.287199 4700 raft_consensus.cc:2932] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:56.289974 8272 tablet_bootstrap.cc:492] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:56.290148 8272 ts_tablet_manager.cc:1367] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20240503 08:12:56.291014 8272 raft_consensus.cc:348] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.291213 8272 raft_consensus.cc:374] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.291270 8272 raft_consensus.cc:729] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.291657 1943 raft_consensus.cc:1203] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:56.291745 8272 consensus_queue.cc:260] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.291945 7737 raft_consensus.cc:482] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:56.292043 7737 raft_consensus.cc:504] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.292330 8260 tablet_bootstrap.cc:492] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:56.292703 7737 leader_election.cc:288] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:56.292915 8260 ts_tablet_manager.cc:1367] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.250s user 0.005s sys 0.000s
I20240503 08:12:56.293027 8254 tablet_bootstrap.cc:492] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:56.293195 8254 ts_tablet_manager.cc:1367] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.301s user 0.039s sys 0.064s
I20240503 08:12:56.293996 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b0c4a0b0564a279806d3fd6f981eed" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
W20240503 08:12:56.294780 1747 leader_election.cc:341] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:56.296638 8272 ts_tablet_manager.cc:1398] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 08:12:56.296950 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b0c4a0b0564a279806d3fd6f981eed" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:56.297634 8260 raft_consensus.cc:348] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.298326 8254 raft_consensus.cc:348] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.298170 8243 tablet_copy_client.cc:308] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.196:42183
I20240503 08:12:56.298638 8254 raft_consensus.cc:374] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.298743 8254 raft_consensus.cc:729] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.299064 8260 raft_consensus.cc:374] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.299432 2224 tablet_copy_service.cc:140] P 184c543cb05d49cc98ba1c56617be7a5: Received BeginTabletCopySession request for tablet df06752e8d7247dd962a376c629473c3 from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:51901)
I20240503 08:12:56.299579 2224 tablet_copy_service.cc:161] P 184c543cb05d49cc98ba1c56617be7a5: Beginning new tablet copy session on tablet df06752e8d7247dd962a376c629473c3 from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:51901: session id = 31f41542452b4b1eb994ff4223d039fc-df06752e8d7247dd962a376c629473c3
I20240503 08:12:56.299639 8260 raft_consensus.cc:729] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.300973 8260 consensus_queue.cc:260] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.302405 2224 tablet_copy_source_session.cc:205] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:56.302927 8260 ts_tablet_manager.cc:1398] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 08:12:56.303272 8260 tablet_bootstrap.cc:492] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:56.304062 8243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df06752e8d7247dd962a376c629473c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.306988 8275 tablet_bootstrap.cc:492] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:56.307175 8275 ts_tablet_manager.cc:1367] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.000s
I20240503 08:12:56.307148 8254 consensus_queue.cc:260] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.308058 8275 raft_consensus.cc:348] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.308293 8275 raft_consensus.cc:374] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.308387 8275 raft_consensus.cc:729] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.308621 8275 consensus_queue.cc:260] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.309214 8275 ts_tablet_manager.cc:1398] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:12:56.309379 8271 ts_tablet_manager.cc:1398] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.063s user 0.000s sys 0.002s
I20240503 08:12:56.309414 7964 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:56.310791 7941 raft_consensus.cc:482] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:56.310948 7941 raft_consensus.cc:504] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.311726 7941 leader_election.cc:288] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:56.313162 8254 ts_tablet_manager.cc:1398] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240503 08:12:56.314061 8261 tablet_bootstrap.cc:492] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
W20240503 08:12:56.297605 1749 leader_election.cc:341] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:56.314296 8261 ts_tablet_manager.cc:1367] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.000s
I20240503 08:12:56.314394 1749 leader_election.cc:302] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f581390da449ccb9e299011296405e; no voters: 31f41542452b4b1eb994ff4223d039fc, 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:12:56.314888 7737 raft_consensus.cc:2726] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:56.315441 8261 raft_consensus.cc:348] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.315770 8261 raft_consensus.cc:374] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.315881 8261 raft_consensus.cc:729] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.316258 8261 consensus_queue.cc:260] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.317529 1812 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:56.319590 8275 tablet_bootstrap.cc:492] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:56.321085 8261 ts_tablet_manager.cc:1398] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:56.326330 1665 catalog_manager.cc:5484] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 7 to 8, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:56.327679 8050 raft_consensus.cc:1050] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:12:56.327459 4566 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:56.328536 8050 consensus_queue.cc:237] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:56.333019 8275 tablet_bootstrap.cc:654] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.335590 4967 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:56.337056 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a31c72b58c4abaaeeb4cf0ec6e6a02" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:56.337394 4963 raft_consensus.cc:2454] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:56.338045 1878 leader_election.cc:302] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:56.338454 7964 raft_consensus.cc:2781] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:56.338586 7964 raft_consensus.cc:482] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:56.338683 7964 raft_consensus.cc:3037] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.339021 8162 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:56.339584 8162 tablet_bootstrap.cc:492] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:12:56.341910 7964 raft_consensus.cc:504] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.344074 7964 leader_election.cc:288] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:56.344398 8162 ts_tablet_manager.cc:1367] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.683s user 0.045s sys 0.017s
I20240503 08:12:56.344664 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a31c72b58c4abaaeeb4cf0ec6e6a02" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:56.344964 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a31c72b58c4abaaeeb4cf0ec6e6a02" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:56.345181 4963 raft_consensus.cc:3037] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.345407 4700 raft_consensus.cc:3037] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.348806 4700 raft_consensus.cc:2454] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:56.349485 1880 leader_election.cc:302] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:12:56.350126 7964 raft_consensus.cc:2781] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:56.350422 7964 raft_consensus.cc:686] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:56.350831 7964 consensus_queue.cc:237] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.348871 8162 raft_consensus.cc:348] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:56.351809 8162 raft_consensus.cc:729] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:12:56.352048 8162 consensus_queue.cc:260] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:12:56.352870 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a31c72b58c4abaaeeb4cf0ec6e6a02" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:56.353145 4698 raft_consensus.cc:2362] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1. Re-sending same reply.
I20240503 08:12:56.360159 8162 ts_tablet_manager.cc:1398] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240503 08:12:56.360939 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-1cf9b2eef42e4fff8cdf5f9dd7b8b31e received from {username='slave'} at 127.27.157.199:51509
I20240503 08:12:56.361125 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session f24648c92c384f1ba349eee801f985e2-1cf9b2eef42e4fff8cdf5f9dd7b8b31e on tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:12:56.370131 4963 raft_consensus.cc:2454] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:56.383049 8275 tablet_bootstrap.cc:492] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:56.386170 8275 ts_tablet_manager.cc:1367] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.002s
I20240503 08:12:56.387037 8260 tablet_bootstrap.cc:654] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.387015 8275 raft_consensus.cc:348] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:56.387463 8275 raft_consensus.cc:374] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.387614 8275 raft_consensus.cc:729] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.388267 8275 consensus_queue.cc:260] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:56.388999 8275 ts_tablet_manager.cc:1398] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:12:56.391463 8201 tablet_copy_client.cc:308] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:12:56.392482 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet 09a268c672ba4580a2d93b5d15d12666 from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:40265)
I20240503 08:12:56.392678 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet 09a268c672ba4580a2d93b5d15d12666 from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:40265: session id = 184c543cb05d49cc98ba1c56617be7a5-09a268c672ba4580a2d93b5d15d12666
I20240503 08:12:56.393570 8247 raft_consensus.cc:482] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:56.393805 8247 raft_consensus.cc:504] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.394873 8247 leader_election.cc:288] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:56.395926 4988 tablet_copy_source_session.cc:205] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:56.396755 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438146d5fa194156b929d7190b11a5de" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:56.397009 2072 raft_consensus.cc:2454] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:56.398329 4568 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:12:56.396574 4566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438146d5fa194156b929d7190b11a5de" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:56.398540 4566 raft_consensus.cc:2454] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:56.399008 4900 leader_election.cc:302] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:56.399552 8064 raft_consensus.cc:2781] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:56.399677 8064 raft_consensus.cc:482] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:56.399770 8064 raft_consensus.cc:3037] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.400401 2072 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:12:56.401623 8201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a268c672ba4580a2d93b5d15d12666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.401908 7737 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:56.402562 7737 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:56.407991 4963 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:12:56.408766 8050 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:56.410398 8064 raft_consensus.cc:504] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.411104 8064 leader_election.cc:288] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:12:56.411752 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438146d5fa194156b929d7190b11a5de" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:56.412045 2073 raft_consensus.cc:3037] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.412235 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438146d5fa194156b929d7190b11a5de" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:56.412573 4568 raft_consensus.cc:3037] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.417611 2073 raft_consensus.cc:2454] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:56.418134 4901 leader_election.cc:302] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:12:56.420163 8246 raft_consensus.cc:2781] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:56.420367 8246 raft_consensus.cc:686] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:56.420779 8246 consensus_queue.cc:237] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:56.427572 4568 raft_consensus.cc:2454] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:56.429684 8201 tablet_copy_client.cc:783] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:56.429978 8201 tablet_copy_client.cc:647] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:56.409400 8243 tablet_copy_client.cc:783] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:56.437208 8243 tablet_copy_client.cc:647] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:56.444593 8247 raft_consensus.cc:1050] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:12:56.445596 8247 consensus_queue.cc:237] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:56.448690 8243 tablet_copy_client.cc:523] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:56.452688 8050 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:56.453898 8243 tablet_bootstrap.cc:492] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:56.459951 4566 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:56.464674 2072 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:56.460299 1668 catalog_manager.cc:5484] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:56.474149 4834 consensus_queue.cc:237] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:56.483690 4963 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
W20240503 08:12:56.485409 4769 consensus_peers.cc:485] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98ad826dca7c4343afcae34bf1a40da8. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:56.486567 4700 raft_consensus.cc:1261] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:56.487740 8161 consensus_queue.cc:1035] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:56.489558 1667 catalog_manager.cc:5484] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:56.498042 1670 catalog_manager.cc:5484] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:56.501754 4568 raft_consensus.cc:1261] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:56.504468 1941 consensus_queue.cc:237] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:12:56.509433 4967 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:12:56.510161 7964 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:12:56.510530 8260 tablet_bootstrap.cc:492] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:56.510756 8260 ts_tablet_manager.cc:1367] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.208s user 0.000s sys 0.006s
I20240503 08:12:56.512739 4700 raft_consensus.cc:1261] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:56.512919 1812 raft_consensus.cc:1261] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:56.513594 8260 raft_consensus.cc:348] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.513703 8247 consensus_queue.cc:1035] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:56.514035 8260 raft_consensus.cc:374] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.514209 8260 raft_consensus.cc:729] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.514161 8064 consensus_queue.cc:1035] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:56.515676 8260 consensus_queue.cc:260] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.518182 8260 ts_tablet_manager.cc:1398] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:56.518692 8260 tablet_bootstrap.cc:492] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:56.520126 2201 raft_consensus.cc:1261] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:56.521574 8201 tablet_copy_client.cc:523] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:56.523638 8246 consensus_queue.cc:1035] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:56.507650 4565 raft_consensus.cc:1261] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:12:56.527150 7941 consensus_queue.cc:1035] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:12:56.527846 7941 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:56.532328 8161 consensus_queue.cc:1035] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:56.533668 7742 raft_consensus.cc:2932] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:56.533980 4965 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:56.544909 4698 raft_consensus.cc:2932] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:56.550397 1667 catalog_manager.cc:5484] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:56.553619 4835 raft_consensus.cc:1203] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:12:56.556149 8028 raft_consensus.cc:482] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:56.556315 8111 raft_consensus.cc:482] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:56.556349 8028 raft_consensus.cc:504] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.556540 8111 raft_consensus.cc:504] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.557593 8111 leader_election.cc:288] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:56.557649 4568 raft_consensus.cc:2932] T 53197deeb8d7439c90b019a6a90857e0 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:12:56.557994 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b0c4a0b0564a279806d3fd6f981eed" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:56.558204 1812 raft_consensus.cc:2454] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:56.558355 8028 leader_election.cc:288] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:56.558737 2009 leader_election.cc:302] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:56.559548 7922 raft_consensus.cc:2781] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:56.559669 7922 raft_consensus.cc:482] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:56.559765 7922 raft_consensus.cc:3037] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.560410 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b0c4a0b0564a279806d3fd6f981eed" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:56.560602 1812 raft_consensus.cc:2454] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:56.561945 4637 leader_election.cc:302] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:56.562498 7922 raft_consensus.cc:504] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.562654 8182 raft_consensus.cc:2781] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:56.562826 8182 raft_consensus.cc:482] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:56.562956 8182 raft_consensus.cc:3037] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.564095 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b0c4a0b0564a279806d3fd6f981eed" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:56.564325 2072 raft_consensus.cc:2379] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31f41542452b4b1eb994ff4223d039fc in current term 1: Already voted for candidate 43e2e01cdd314f9c872c4336ef74250a in this term.
I20240503 08:12:56.564879 8260 tablet_bootstrap.cc:654] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:56.565274 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b0c4a0b0564a279806d3fd6f981eed" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:56.565922 7922 leader_election.cc:288] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:56.566402 8182 raft_consensus.cc:504] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.567628 4699 raft_consensus.cc:2379] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43e2e01cdd314f9c872c4336ef74250a in current term 1: Already voted for candidate 31f41542452b4b1eb994ff4223d039fc in this term.
I20240503 08:12:56.570190 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53197deeb8d7439c90b019a6a90857e0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:56.571669 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b0c4a0b0564a279806d3fd6f981eed" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:56.571863 1812 raft_consensus.cc:3037] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.574875 1812 raft_consensus.cc:2454] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:56.576623 1667 catalog_manager.cc:5484] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 8 to 9, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:56.569047 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b0c4a0b0564a279806d3fd6f981eed" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:56.581465 2072 raft_consensus.cc:2379] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 31f41542452b4b1eb994ff4223d039fc in current term 1: Already voted for candidate 43e2e01cdd314f9c872c4336ef74250a in this term.
I20240503 08:12:56.568027 8182 leader_election.cc:288] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:56.594666 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b0c4a0b0564a279806d3fd6f981eed" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:56.595822 4699 raft_consensus.cc:2379] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 43e2e01cdd314f9c872c4336ef74250a in current term 1: Already voted for candidate 31f41542452b4b1eb994ff4223d039fc in this term.
I20240503 08:12:56.581004 8312 ts_tablet_manager.cc:902] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:56.587154 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet 53197deeb8d7439c90b019a6a90857e0 with delete_type TABLET_DATA_TOMBSTONED (TS b8f581390da449ccb9e299011296405e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:56.593853 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12b0c4a0b0564a279806d3fd6f981eed" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:56.601048 8243 tablet_bootstrap.cc:492] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:56.604230 8243 tablet_bootstrap.cc:492] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:12:56.604872 8243 ts_tablet_manager.cc:1367] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.151s user 0.002s sys 0.011s
I20240503 08:12:56.605288 2009 leader_election.cc:302] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters: 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:56.606405 8243 raft_consensus.cc:348] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:56.606842 8243 raft_consensus.cc:729] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:12:56.607367 8243 consensus_queue.cc:260] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:12:56.608024 8028 raft_consensus.cc:2781] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:56.608291 8028 raft_consensus.cc:686] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:56.608664 8028 consensus_queue.cc:237] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:56.608983 8243 ts_tablet_manager.cc:1398] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 08:12:56.619261 8201 tablet_bootstrap.cc:492] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:56.621281 8316 tablet_replica.cc:330] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:56.621619 8316 raft_consensus.cc:2229] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:56.622011 8316 raft_consensus.cc:2258] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:56.623201 8316 ts_tablet_manager.cc:1875] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:56.625905 8041 raft_consensus.cc:2932] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:56.631963 4700 raft_consensus.cc:2932] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:56.632347 1812 raft_consensus.cc:2932] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:56.636139 8316 ts_tablet_manager.cc:1888] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:12:56.636346 8316 log.cc:1196] T 53197deeb8d7439c90b019a6a90857e0 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/53197deeb8d7439c90b019a6a90857e0
I20240503 08:12:56.637449 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 53197deeb8d7439c90b019a6a90857e0 (table rebalance_test_table_15 [id=c9883eb3c54f4eb89c079762678d5d4a]) successfully deleted
I20240503 08:12:56.638958 2201 raft_consensus.cc:2932] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:56.642465 1665 catalog_manager.cc:5484] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:12:56.643199 4636 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.196:42183, user_credentials={real_user=slave}} blocked reactor thread for 71811.2us
I20240503 08:12:56.643841 2224 tablet_copy_service.cc:342] P 184c543cb05d49cc98ba1c56617be7a5: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-df06752e8d7247dd962a376c629473c3 received from {username='slave'} at 127.27.157.198:51901
I20240503 08:12:56.643970 2224 tablet_copy_service.cc:433] P 184c543cb05d49cc98ba1c56617be7a5: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-df06752e8d7247dd962a376c629473c3 on tablet df06752e8d7247dd962a376c629473c3 with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:56.646860 4636 leader_election.cc:302] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc; no voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e
I20240503 08:12:56.647186 8311 raft_consensus.cc:2726] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 08:12:56.654662 1667 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:56.672320 4967 consensus_queue.cc:237] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:56.680313 2204 raft_consensus.cc:1261] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:56.683753 8246 consensus_queue.cc:1035] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:56.688628 1813 consensus_queue.cc:237] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:12:56.688628 8227 raft_consensus.cc:2932] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:56.693069 4963 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:12:56.693775 7737 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:12:56.694267 4566 raft_consensus.cc:1261] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:12:56.695443 7737 consensus_queue.cc:1035] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:12:56.697424 4700 raft_consensus.cc:1261] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:56.697656 4963 raft_consensus.cc:1261] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:56.698459 7941 consensus_queue.cc:1035] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.699544 2204 raft_consensus.cc:2932] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:56.701620 7941 consensus_queue.cc:1035] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.710433 4566 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:56.722611 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a0bffb0aa304bd184b8ad26bc94fd18 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:56.723505 4965 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:56.727252 1667 catalog_manager.cc:5484] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:56.741257 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0bffb0aa304bd184b8ad26bc94fd18 with delete_type TABLET_DATA_TOMBSTONED (TS 31f41542452b4b1eb994ff4223d039fc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:56.742976 8328 tablet_replica.cc:330] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:56.745065 8328 raft_consensus.cc:2229] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:56.746361 8328 raft_consensus.cc:2258] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:56.747311 8300 raft_consensus.cc:2932] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:56.748683 8328 ts_tablet_manager.cc:1875] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:56.762156 8328 ts_tablet_manager.cc:1888] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:56.771975 4698 raft_consensus.cc:1203] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:56.776789 8328 log.cc:1196] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/9a0bffb0aa304bd184b8ad26bc94fd18
I20240503 08:12:56.778760 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 9a0bffb0aa304bd184b8ad26bc94fd18 (table rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]) successfully deleted
I20240503 08:12:56.784617 1812 raft_consensus.cc:1261] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:56.785370 8247 consensus_queue.cc:1035] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:56.787315 1812 raft_consensus.cc:2932] T 9a0bffb0aa304bd184b8ad26bc94fd18 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:56.795989 1670 catalog_manager.cc:5484] T 3219159f92774adbb7ecaa77ac267021 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 8 to 9, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:12:56.801301 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3219159f92774adbb7ecaa77ac267021 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:56.801641 8201 tablet_bootstrap.cc:700] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5: Time spent opening tablet: real 0.179s user 0.002s sys 0.000s
I20240503 08:12:56.809427 8247 consensus_queue.cc:1035] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.819218 8161 raft_consensus.cc:1050] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a to VOTER
I20240503 08:12:56.820175 8161 consensus_queue.cc:237] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:12:56.824874 8320 raft_consensus.cc:482] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:56.825070 8320 raft_consensus.cc:504] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:56.825276 1941 raft_consensus.cc:1261] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:56.825912 8320 leader_election.cc:288] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:56.826128 8065 consensus_queue.cc:1035] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:56.827042 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3070c262ed420a91ac0ed063d16471" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:56.827291 1812 raft_consensus.cc:2454] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:56.828070 2009 leader_election.cc:302] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:56.828522 7922 raft_consensus.cc:2781] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:56.828631 7922 raft_consensus.cc:482] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:56.828734 7922 raft_consensus.cc:3037] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:56.830691 1812 raft_consensus.cc:1261] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:56.831295 4566 raft_consensus.cc:1261] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:56.831341 7742 consensus_queue.cc:1035] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:56.831902 7742 consensus_queue.cc:1035] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:56.831923 8247 consensus_queue.cc:1035] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.832486 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3070c262ed420a91ac0ed063d16471" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
W20240503 08:12:56.833253 2010 leader_election.cc:341] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:56.835983 7922 raft_consensus.cc:504] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:56.837617 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3070c262ed420a91ac0ed063d16471" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:56.837711 7922 leader_election.cc:288] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:56.838157 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3070c262ed420a91ac0ed063d16471" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:56.838397 1813 raft_consensus.cc:3037] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:12:56.838618 2010 leader_election.cc:341] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:56.842051 1813 raft_consensus.cc:2454] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:56.843436 2009 leader_election.cc:302] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e; no voters: 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:56.844574 2072 raft_consensus.cc:1203] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:56.845048 7922 raft_consensus.cc:2781] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:56.845422 7922 raft_consensus.cc:686] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:56.845808 7922 consensus_queue.cc:237] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:56.850898 8312 tablet_copy_client.cc:308] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:12:56.851912 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet 929d8d354b914502b5e677ccc84e2eda from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:54589)
I20240503 08:12:56.852102 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet 929d8d354b914502b5e677ccc84e2eda from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:54589: session id = b8f581390da449ccb9e299011296405e-929d8d354b914502b5e677ccc84e2eda
I20240503 08:12:56.855574 4855 tablet_copy_source_session.cc:205] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:56.861184 8065 raft_consensus.cc:2932] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:56.862643 4569 raft_consensus.cc:2932] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:56.863955 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929d8d354b914502b5e677ccc84e2eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.874689 8260 tablet_bootstrap.cc:492] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:56.874922 8260 ts_tablet_manager.cc:1367] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.356s user 0.014s sys 0.038s
I20240503 08:12:56.854544 4565 raft_consensus.cc:2932] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:56.875450 1813 raft_consensus.cc:2932] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:56.886911 1668 catalog_manager.cc:5484] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:56.916173 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet 3219159f92774adbb7ecaa77ac267021 with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:56.916801 1941 raft_consensus.cc:2932] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:56.919430 8345 ts_tablet_manager.cc:902] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:56.939425 4834 consensus_queue.cc:237] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:12:56.942150 1813 raft_consensus.cc:1261] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:56.943584 8065 consensus_queue.cc:1035] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:56.947237 1941 raft_consensus.cc:1261] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:56.947566 7742 raft_consensus.cc:2932] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:56.948135 1813 raft_consensus.cc:2932] T 1e641fe703c14f4cb1bd22c676d79576 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:56.949580 8260 raft_consensus.cc:348] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:56.949927 8260 raft_consensus.cc:374] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:56.950094 8260 raft_consensus.cc:729] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:56.952603 7742 consensus_queue.cc:1035] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:56.954922 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1e641fe703c14f4cb1bd22c676d79576 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:56.956233 8345 tablet_copy_client.cc:308] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:12:56.957284 1943 raft_consensus.cc:2932] T 1e641fe703c14f4cb1bd22c676d79576 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:56.958699 1669 catalog_manager.cc:5484] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:56.970081 4547 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e641fe703c14f4cb1bd22c676d79576 with delete_type TABLET_DATA_TOMBSTONED (TS 5f95b05d111e4090bbc59dec394ed0ab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:56.974714 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet 98ad826dca7c4343afcae34bf1a40da8 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a ({username='slave'} at 127.27.157.194:55239)
I20240503 08:12:56.974984 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet 98ad826dca7c4343afcae34bf1a40da8 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a at {username='slave'} at 127.27.157.194:55239: session id = 315f3f0bcbf447a4b93b1aafcfbf0f5a-98ad826dca7c4343afcae34bf1a40da8
I20240503 08:12:56.976688 8260 consensus_queue.cc:260] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:56.978389 4855 tablet_copy_source_session.cc:205] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:12:56.979844 8345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ad826dca7c4343afcae34bf1a40da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:56.991951 8345 tablet_copy_client.cc:783] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:56.992498 8345 tablet_copy_client.cc:647] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:56.994450 8201 tablet_bootstrap.cc:492] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:56.994828 8345 tablet_copy_client.cc:523] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:56.994984 8201 tablet_bootstrap.cc:492] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:12:56.995316 8201 ts_tablet_manager.cc:1367] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.376s user 0.010s sys 0.000s
I20240503 08:12:57.001505 8345 tablet_bootstrap.cc:492] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:57.004523 8260 ts_tablet_manager.cc:1398] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.129s user 0.004s sys 0.000s
I20240503 08:12:57.017115 1669 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 (table rebalancer_dynamic_table [id=163bce7cc559485da3b201768dbcf989]). Expected version 1 got 0
I20240503 08:12:56.977994 1670 catalog_manager.cc:5484] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:57.029356 8345 tablet_bootstrap.cc:492] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:57.030483 8345 tablet_bootstrap.cc:492] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap complete.
I20240503 08:12:57.031415 8345 ts_tablet_manager.cc:1367] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.009s
I20240503 08:12:57.031598 7922 consensus_queue.cc:1035] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:56.989923 8346 tablet_replica.cc:330] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:57.032723 8345 raft_consensus.cc:348] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:57.034411 8345 raft_consensus.cc:729] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: LEARNER
I20240503 08:12:57.034821 8345 consensus_queue.cc:260] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:12:57.037447 8345 ts_tablet_manager.cc:1398] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 08:12:57.038309 1669 catalog_manager.cc:5484] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:57.039217 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-98ad826dca7c4343afcae34bf1a40da8 received from {username='slave'} at 127.27.157.194:55239
I20240503 08:12:57.039443 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-98ad826dca7c4343afcae34bf1a40da8 on tablet 98ad826dca7c4343afcae34bf1a40da8 with peer 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:12:57.045246 8346 raft_consensus.cc:2229] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:57.047753 8346 raft_consensus.cc:2258] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.054863 8346 ts_tablet_manager.cc:1875] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:57.063730 8359 tablet.cc:1722] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:57.064878 8359 tablet.cc:1620] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:57.064813 7922 consensus_queue.cc:1035] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:57.078675 8312 tablet_copy_client.cc:783] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:12:57.079068 8312 tablet_copy_client.cc:647] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:57.087082 1667 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=163bce7cc559485da3b201768dbcf989] alter complete (version 1)
I20240503 08:12:57.096836 8358 tablet.cc:1722] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:57.097577 8358 tablet.cc:1620] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: MemRowSet was empty: no flush needed.
I20240503 08:12:57.098330 8201 raft_consensus.cc:348] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:57.098620 8201 raft_consensus.cc:729] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:12:57.098825 8201 consensus_queue.cc:260] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:57.101019 8201 ts_tablet_manager.cc:1398] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.106s user 0.002s sys 0.000s
I20240503 08:12:57.102097 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-09a268c672ba4580a2d93b5d15d12666 received from {username='slave'} at 127.27.157.196:40265
I20240503 08:12:57.102309 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-09a268c672ba4580a2d93b5d15d12666 on tablet 09a268c672ba4580a2d93b5d15d12666 with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:57.105525 7964 raft_consensus.cc:1050] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:12:57.106475 7964 consensus_queue.cc:237] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:57.108016 1665 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 (table rebalancer_dynamic_table [id=163bce7cc559485da3b201768dbcf989]). Expected version 1 got 0
I20240503 08:12:57.109756 8312 tablet_copy_client.cc:523] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:57.113905 8355 tablet_replica.cc:330] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:57.115456 8355 raft_consensus.cc:2229] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:57.116997 8355 raft_consensus.cc:2258] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.119869 8312 tablet_bootstrap.cc:492] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:57.122603 8365 tablet.cc:1722] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:57.123591 8355 ts_tablet_manager.cc:1875] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:57.126796 1669 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 (table rebalancer_dynamic_table [id=163bce7cc559485da3b201768dbcf989]). Expected version 1 got 0
I20240503 08:12:57.130121 8365 tablet.cc:1620] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:57.137454 4965 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:12:57.137475 1813 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:12:57.138067 7941 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:57.138362 7964 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:57.139801 4834 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:12:57.140481 7941 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:12:57.154067 8346 ts_tablet_manager.cc:1888] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:12:57.164986 8253 raft_consensus.cc:1050] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:12:57.165905 8253 consensus_queue.cc:237] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:57.173887 7964 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:57.177898 4965 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:57.178023 4835 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:57.182287 8346 log.cc:1196] T 3219159f92774adbb7ecaa77ac267021 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/3219159f92774adbb7ecaa77ac267021
I20240503 08:12:57.183461 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 3219159f92774adbb7ecaa77ac267021 (table rebalance_test_table_1 [id=9653225381b14c348f664a692ff48da1]) successfully deleted
I20240503 08:12:57.184372 1813 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:57.188689 1667 catalog_manager.cc:5484] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:57.191802 1669 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:57.204128 1668 catalog_manager.cc:5576] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559) does not have the latest schema for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 (table rebalancer_dynamic_table [id=163bce7cc559485da3b201768dbcf989]). Expected version 2 got 1
I20240503 08:12:57.208525 8358 tablet.cc:1722] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:57.211690 8358 tablet.cc:1620] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: MemRowSet was empty: no flush needed.
I20240503 08:12:57.217111 8359 tablet.cc:1722] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:57.217732 8359 tablet.cc:1620] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:57.224534 8355 ts_tablet_manager.cc:1888] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:57.227037 8355 log.cc:1196] T 1e641fe703c14f4cb1bd22c676d79576 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/1e641fe703c14f4cb1bd22c676d79576
I20240503 08:12:57.233114 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 1e641fe703c14f4cb1bd22c676d79576 (table rebalancer_extra_table_12 [id=b06077fc3c724567becc26a279a4b8ec]) successfully deleted
I20240503 08:12:57.248471 8314 raft_consensus.cc:1050] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc: attempting to promote NON_VOTER 43e2e01cdd314f9c872c4336ef74250a to VOTER
I20240503 08:12:57.249367 8314 consensus_queue.cc:237] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:12:57.254714 1812 raft_consensus.cc:1261] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:57.254765 2071 raft_consensus.cc:1261] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:57.255429 8311 consensus_queue.cc:1035] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:57.255961 8373 consensus_queue.cc:1035] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:57.257666 4700 raft_consensus.cc:1261] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:57.259999 4963 raft_consensus.cc:1261] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:57.260591 8314 raft_consensus.cc:482] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:57.260840 8314 raft_consensus.cc:504] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:12:57.260861 8027 consensus_queue.cc:1035] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:57.262336 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3070c262ed420a91ac0ed063d16471" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:57.263057 8314 leader_election.cc:288] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:57.263157 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3070c262ed420a91ac0ed063d16471" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:57.263459 8027 consensus_queue.cc:1035] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:57.263706 1813 raft_consensus.cc:2379] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31f41542452b4b1eb994ff4223d039fc in current term 1: Already voted for candidate 43e2e01cdd314f9c872c4336ef74250a in this term.
I20240503 08:12:57.264250 4637 leader_election.cc:302] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc; no voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e
I20240503 08:12:57.264709 8311 raft_consensus.cc:3037] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.273960 4565 raft_consensus.cc:1261] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:57.275566 8311 raft_consensus.cc:2726] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:57.276189 8027 consensus_queue.cc:1035] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:57.284761 4965 raft_consensus.cc:1261] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:57.287123 8311 consensus_queue.cc:1035] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:57.292919 8253 raft_consensus.cc:2932] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:57.293707 4967 raft_consensus.cc:2932] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:57.293804 1670 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=163bce7cc559485da3b201768dbcf989] alter complete (version 2)
I20240503 08:12:57.303916 8312 tablet_bootstrap.cc:492] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:57.304482 8312 tablet_bootstrap.cc:492] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:12:57.303989 1667 catalog_manager.cc:5484] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 2 to 3, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:57.304893 8312 ts_tablet_manager.cc:1367] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.185s user 0.009s sys 0.003s
I20240503 08:12:57.311753 8312 raft_consensus.cc:348] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:57.311586 4699 raft_consensus.cc:2932] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:57.312134 8312 raft_consensus.cc:729] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:12:57.312352 8312 consensus_queue.cc:260] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:57.314297 8312 ts_tablet_manager.cc:1398] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:12:57.316836 8365 tablet.cc:1722] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:57.325112 2204 raft_consensus.cc:1203] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:57.329105 4568 raft_consensus.cc:2932] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:57.334285 4963 raft_consensus.cc:2932] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:57.332566 8311 raft_consensus.cc:2932] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:57.337929 8365 tablet.cc:1620] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: MemRowSet was empty: no flush needed.
I20240503 08:12:57.340792 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session b8f581390da449ccb9e299011296405e-929d8d354b914502b5e677ccc84e2eda received from {username='slave'} at 127.27.157.193:54589
I20240503 08:12:57.340953 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session b8f581390da449ccb9e299011296405e-929d8d354b914502b5e677ccc84e2eda on tablet 929d8d354b914502b5e677ccc84e2eda with peer b8f581390da449ccb9e299011296405e
I20240503 08:12:57.345091 1812 raft_consensus.cc:2932] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:57.348387 2072 raft_consensus.cc:2932] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:57.352785 1665 catalog_manager.cc:5484] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:57.360971 2204 consensus_queue.cc:237] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:12:57.364274 4565 raft_consensus.cc:1261] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:57.364365 4700 raft_consensus.cc:1261] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:57.364960 8027 consensus_queue.cc:1035] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:57.366358 8027 consensus_queue.cc:1035] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:57.370110 7924 raft_consensus.cc:2932] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:57.374794 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df06752e8d7247dd962a376c629473c3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:57.375079 4565 raft_consensus.cc:2932] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:57.375623 4700 raft_consensus.cc:2932] T df06752e8d7247dd962a376c629473c3 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:57.385141 1670 catalog_manager.cc:5484] T df06752e8d7247dd962a376c629473c3 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:12:57.398800 4945 tablet_service.cc:1514] Processing DeleteTablet for tablet df06752e8d7247dd962a376c629473c3 with delete_type TABLET_DATA_TOMBSTONED (TS 00b61625d8a7460892c118a2570afda5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:57.400279 8028 consensus_queue.cc:1035] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:57.409637 4700 consensus_queue.cc:237] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:12:57.412590 8385 tablet_replica.cc:330] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:57.412999 2071 raft_consensus.cc:1261] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:57.413779 8314 consensus_queue.cc:1035] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:57.414295 1670 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:57.418201 8385 raft_consensus.cc:2229] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:57.418937 8373 raft_consensus.cc:2932] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:57.421058 2071 raft_consensus.cc:2932] T 0caddf5e39f8440baa8ad3656ce91a8e P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:57.431416 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0caddf5e39f8440baa8ad3656ce91a8e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:57.429351 4965 raft_consensus.cc:1261] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:57.433648 8311 consensus_queue.cc:1035] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:57.433805 8385 raft_consensus.cc:2258] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.435773 4965 raft_consensus.cc:2932] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:57.437340 8358 tablet.cc:1722] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:57.438321 8365 tablet.cc:1722] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:57.444590 8359 tablet.cc:1722] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:57.448493 1668 catalog_manager.cc:5484] T 0caddf5e39f8440baa8ad3656ce91a8e P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:12:57.451170 8028 consensus_queue.cc:1035] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:57.455914 8385 ts_tablet_manager.cc:1875] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:57.464262 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet 0caddf5e39f8440baa8ad3656ce91a8e with delete_type TABLET_DATA_TOMBSTONED (TS b8f581390da449ccb9e299011296405e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48256
I20240503 08:12:57.465355 8391 tablet_replica.cc:330] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:12:57.469453 8391 raft_consensus.cc:2229] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:57.470633 8391 raft_consensus.cc:2258] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.474207 1663 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=163bce7cc559485da3b201768dbcf989] alter complete (version 3)
I20240503 08:12:57.477424 8391 ts_tablet_manager.cc:1875] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:57.476845 1812 raft_consensus.cc:1203] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:57.482367 8385 ts_tablet_manager.cc:1888] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:57.482539 8385 log.cc:1196] T df06752e8d7247dd962a376c629473c3 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/df06752e8d7247dd962a376c629473c3
I20240503 08:12:57.483662 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet df06752e8d7247dd962a376c629473c3 (table rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]) successfully deleted
I20240503 08:12:57.485304 1943 raft_consensus.cc:1203] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:12:57.510004 8391 ts_tablet_manager.cc:1888] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:57.531446 8391 log.cc:1196] T 0caddf5e39f8440baa8ad3656ce91a8e P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/0caddf5e39f8440baa8ad3656ce91a8e
I20240503 08:12:57.536513 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 0caddf5e39f8440baa8ad3656ce91a8e (table rebalancer_extra_table_12 [id=b06077fc3c724567becc26a279a4b8ec]) successfully deleted
I20240503 08:12:57.560889 1670 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:57.575748 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 541925a798914d578021ce1f0d166730 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=f0aa7c71965d48539c0ac7118319411e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.576212 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 541925a798914d578021ce1f0d166730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.576319 4813 tablet_service.cc:1467] Processing CreateTablet for tablet d6b22a007a2540ea9364579ac680fcf4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=f0aa7c71965d48539c0ac7118319411e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.576790 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b22a007a2540ea9364579ac680fcf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.576980 1793 tablet_service.cc:1467] Processing CreateTablet for tablet d6b22a007a2540ea9364579ac680fcf4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=f0aa7c71965d48539c0ac7118319411e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.577821 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b22a007a2540ea9364579ac680fcf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.579772 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 541925a798914d578021ce1f0d166730 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=f0aa7c71965d48539c0ac7118319411e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.580163 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 541925a798914d578021ce1f0d166730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.583267 4683 tablet_service.cc:1467] Processing CreateTablet for tablet d6b22a007a2540ea9364579ac680fcf4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=f0aa7c71965d48539c0ac7118319411e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.583695 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 541925a798914d578021ce1f0d166730 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=f0aa7c71965d48539c0ac7118319411e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.583822 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b22a007a2540ea9364579ac680fcf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.584156 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 541925a798914d578021ce1f0d166730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.588444 1668 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:57.588762 1668 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:57.592109 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3e931766d44f0aaf902cf0e950f83b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=f0aa7c71965d48539c0ac7118319411e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.592617 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3e931766d44f0aaf902cf0e950f83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.596359 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3e931766d44f0aaf902cf0e950f83b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=f0aa7c71965d48539c0ac7118319411e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.596833 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3e931766d44f0aaf902cf0e950f83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.599820 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3e931766d44f0aaf902cf0e950f83b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=f0aa7c71965d48539c0ac7118319411e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:57.600379 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3e931766d44f0aaf902cf0e950f83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.601218 8400 tablet_bootstrap.cc:492] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:57.603114 8400 tablet_bootstrap.cc:654] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.617517 8400 tablet_bootstrap.cc:492] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:57.617703 8400 ts_tablet_manager.cc:1367] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20240503 08:12:57.617797 1668 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02
I20240503 08:12:57.618464 8400 raft_consensus.cc:348] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:57.618664 8400 raft_consensus.cc:374] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.618743 8400 raft_consensus.cc:729] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.619220 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:57 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:57.620149 4681 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:57 UTC) from {username='slave'} at 127.0.0.1:57094
I20240503 08:12:57.620349 8404 tablet_replica.cc:330] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:57.620584 8404 raft_consensus.cc:2229] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:57.620618 4945 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:57 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:57.621088 8404 raft_consensus.cc:2258] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.621870 8385 tablet_replica.cc:330] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:57.622068 8404 ts_tablet_manager.cc:1875] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:57.622076 8385 raft_consensus.cc:2229] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:57.622287 8385 raft_consensus.cc:2258] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.623492 8385 ts_tablet_manager.cc:1875] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:57.628973 8404 ts_tablet_manager.cc:1888] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:57.629187 8404 log.cc:1196] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/24a31c72b58c4abaaeeb4cf0ec6e6a02
I20240503 08:12:57.629827 8385 ts_tablet_manager.cc:1888] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:57.619019 8400 consensus_queue.cc:260] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:57.630376 8404 ts_tablet_manager.cc:1909] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:57.631798 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 (table rebalancer_dynamic_table_new_name [id=163bce7cc559485da3b201768dbcf989]) successfully deleted
I20240503 08:12:57.632050 8400 ts_tablet_manager.cc:1398] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20240503 08:12:57.632314 8385 log.cc:1196] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/24a31c72b58c4abaaeeb4cf0ec6e6a02
I20240503 08:12:57.632805 8385 ts_tablet_manager.cc:1909] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:57.633955 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 (table rebalancer_dynamic_table_new_name [id=163bce7cc559485da3b201768dbcf989]) successfully deleted
I20240503 08:12:57.715229 8405 tablet_replica.cc:330] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:12:57.717320 8405 raft_consensus.cc:2229] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:57.717658 8405 raft_consensus.cc:2258] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:57.718115 8408 tablet_bootstrap.cc:492] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:57.719713 8402 tablet_bootstrap.cc:492] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:57.721024 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:57.725674 8409 tablet_bootstrap.cc:492] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:57.733980 8401 tablet_bootstrap.cc:492] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:57.735178 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 58b502e473224315806bf90ab972eb82 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9aceb057f2af4ae2a244a43374d2a9a1]), partition=
I20240503 08:12:57.735906 8401 tablet_bootstrap.cc:654] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.735992 8408 tablet_bootstrap.cc:654] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.736104 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58b502e473224315806bf90ab972eb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.738544 8402 tablet_bootstrap.cc:654] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.733393 8405 ts_tablet_manager.cc:1875] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:57.740372 8409 tablet_bootstrap.cc:654] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.742569 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 58b502e473224315806bf90ab972eb82 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9aceb057f2af4ae2a244a43374d2a9a1]), partition=
I20240503 08:12:57.743022 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58b502e473224315806bf90ab972eb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.746649 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 58b502e473224315806bf90ab972eb82 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9aceb057f2af4ae2a244a43374d2a9a1]), partition=
I20240503 08:12:57.747110 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58b502e473224315806bf90ab972eb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.752924 8408 tablet_bootstrap.cc:492] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:57.753262 8409 tablet_bootstrap.cc:492] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:57.753222 8408 ts_tablet_manager.cc:1367] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.004s
I20240503 08:12:57.753716 8409 ts_tablet_manager.cc:1367] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20240503 08:12:57.754449 8409 raft_consensus.cc:348] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:57.754534 8408 raft_consensus.cc:348] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.754765 8408 raft_consensus.cc:374] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.754904 8408 raft_consensus.cc:729] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.755448 8416 tablet_bootstrap.cc:492] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:57.755508 8408 consensus_queue.cc:260] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.757716 8416 tablet_bootstrap.cc:654] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.758582 8409 raft_consensus.cc:374] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.758718 8409 raft_consensus.cc:729] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.759034 8409 consensus_queue.cc:260] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:57.762459 8408 ts_tablet_manager.cc:1398] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 08:12:57.763183 8409 ts_tablet_manager.cc:1398] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 08:12:57.771644 8409 tablet_bootstrap.cc:492] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:57.773124 8408 tablet_bootstrap.cc:492] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:57.773555 8409 tablet_bootstrap.cc:654] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.774703 8405 ts_tablet_manager.cc:1888] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:57.774006 8417 tablet_bootstrap.cc:492] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:57.775597 8401 tablet_bootstrap.cc:492] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:57.775632 8408 tablet_bootstrap.cc:654] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.775758 8401 ts_tablet_manager.cc:1367] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.001s
I20240503 08:12:57.776629 8417 tablet_bootstrap.cc:654] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.776677 8401 raft_consensus.cc:348] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.776892 8401 raft_consensus.cc:374] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.776955 8401 raft_consensus.cc:729] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.777222 8401 consensus_queue.cc:260] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.777849 8401 ts_tablet_manager.cc:1398] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:12:57.778235 8401 tablet_bootstrap.cc:492] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:57.781661 8402 tablet_bootstrap.cc:492] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:57.781962 8402 ts_tablet_manager.cc:1367] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20240503 08:12:57.783051 8402 raft_consensus.cc:348] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.783452 8402 raft_consensus.cc:374] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.783689 8402 raft_consensus.cc:729] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.784143 8402 consensus_queue.cc:260] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.785019 8402 ts_tablet_manager.cc:1398] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:57.785259 8405 log.cc:1196] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/24a31c72b58c4abaaeeb4cf0ec6e6a02
I20240503 08:12:57.786106 8405 ts_tablet_manager.cc:1909] T 24a31c72b58c4abaaeeb4cf0ec6e6a02 P 31f41542452b4b1eb994ff4223d039fc: Deleting consensus metadata
I20240503 08:12:57.789222 8401 tablet_bootstrap.cc:654] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.788058 8402 tablet_bootstrap.cc:492] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:57.791633 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 24a31c72b58c4abaaeeb4cf0ec6e6a02 (table rebalancer_dynamic_table_new_name [id=163bce7cc559485da3b201768dbcf989]) successfully deleted
I20240503 08:12:57.800246 8409 tablet_bootstrap.cc:492] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:57.801517 8409 ts_tablet_manager.cc:1367] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.004s
I20240503 08:12:57.801744 8402 tablet_bootstrap.cc:654] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.802366 8409 raft_consensus.cc:348] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:57.809715 8409 raft_consensus.cc:374] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.809914 8409 raft_consensus.cc:729] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.810245 8409 consensus_queue.cc:260] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:57.811668 8401 tablet_bootstrap.cc:492] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:57.811937 8401 ts_tablet_manager.cc:1367] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.003s
I20240503 08:12:57.813051 8401 raft_consensus.cc:348] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.813407 8401 raft_consensus.cc:374] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.813593 8401 raft_consensus.cc:729] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.814062 8401 consensus_queue.cc:260] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.816560 8401 ts_tablet_manager.cc:1398] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:57.816979 8401 tablet_bootstrap.cc:492] T 541925a798914d578021ce1f0d166730 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:12:57.818542 8416 tablet_bootstrap.cc:492] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:57.818773 8416 ts_tablet_manager.cc:1367] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.003s
I20240503 08:12:57.820032 8416 raft_consensus.cc:348] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:57.821601 8416 raft_consensus.cc:374] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.822038 8416 raft_consensus.cc:729] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.824016 8409 ts_tablet_manager.cc:1398] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240503 08:12:57.825026 8416 consensus_queue.cc:260] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:57.825736 8401 tablet_bootstrap.cc:654] T 541925a798914d578021ce1f0d166730 P b8f581390da449ccb9e299011296405e: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:57.826756 8416 ts_tablet_manager.cc:1398] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 08:12:57.835816 8402 tablet_bootstrap.cc:492] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:57.836140 8402 ts_tablet_manager.cc:1367] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.005s
I20240503 08:12:57.836917 7742 raft_consensus.cc:482] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:57.837147 7742 raft_consensus.cc:504] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.837538 8402 raft_consensus.cc:348] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.837901 8402 raft_consensus.cc:374] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:57.838037 8402 raft_consensus.cc:729] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:57.838212 7742 leader_election.cc:288] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:57.838439 8402 consensus_queue.cc:260] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.839809 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b22a007a2540ea9364579ac680fcf4" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:57.843813 4699 raft_consensus.cc:2454] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 0.
I20240503 08:12:57.839298 8402 ts_tablet_manager.cc:1398] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 08:12:57.849722 4770 leader_election.cc:302] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:57.850325 8161 raft_consensus.cc:2781] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:57.850509 8161 raft_consensus.cc:482] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:57.850623 8161 raft_consensus.cc:3037] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.858872 8360 raft_consensus.cc:482] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:57.859117 8360 raft_consensus.cc:504] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:57.860215 8360 leader_election.cc:288] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:57.864676 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541925a798914d578021ce1f0d166730" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:57.865056 2204 raft_consensus.cc:2454] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:57.884240 1878 leader_election.cc:302] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:57.884765 7964 raft_consensus.cc:2781] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:57.884878 7964 raft_consensus.cc:482] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:57.884969 7964 raft_consensus.cc:3037] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.893486 8111 raft_consensus.cc:482] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:57.893779 8111 raft_consensus.cc:504] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.894747 8111 leader_election.cc:288] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:57.910890 8064 raft_consensus.cc:1050] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:12:57.912252 8064 consensus_queue.cc:237] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:57.912525 7964 raft_consensus.cc:504] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:57.914436 7964 leader_election.cc:288] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:12:57.923269 2201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541925a798914d578021ce1f0d166730" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:57.923887 2201 raft_consensus.cc:3037] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.932173 4568 raft_consensus.cc:1261] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:57.932866 8041 consensus_queue.cc:1035] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:57.945257 2203 raft_consensus.cc:1261] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:57.946089 4834 raft_consensus.cc:1261] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:57.947640 8041 consensus_queue.cc:1035] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:57.949828 8041 consensus_queue.cc:1035] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:57.962307 8161 raft_consensus.cc:504] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:57.963589 8161 leader_election.cc:288] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:12:57.964233 7941 raft_consensus.cc:482] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:57.964426 7941 raft_consensus.cc:504] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:57.965456 7941 leader_election.cc:288] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:57.966291 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b22a007a2540ea9364579ac680fcf4" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:57.966715 4698 raft_consensus.cc:3037] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.969548 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b502e473224315806bf90ab972eb82" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:57.970860 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b502e473224315806bf90ab972eb82" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:12:57.971258 4963 raft_consensus.cc:2454] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 0.
I20240503 08:12:57.972191 1878 leader_election.cc:302] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
W20240503 08:12:57.972596 1878 leader_election.cc:341] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:57.973284 7941 raft_consensus.cc:2781] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:57.973384 7941 raft_consensus.cc:482] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:57.973460 7941 raft_consensus.cc:3037] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:57.971738 4698 raft_consensus.cc:2454] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:57.978106 7941 raft_consensus.cc:504] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:57.988427 2072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b502e473224315806bf90ab972eb82" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:57.990042 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b502e473224315806bf90ab972eb82" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:12:57.990393 4963 raft_consensus.cc:3037] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
W20240503 08:12:58.005632 1878 leader_election.cc:341] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:58.020612 2201 raft_consensus.cc:2454] T 541925a798914d578021ce1f0d166730 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:58.021512 1878 leader_election.cc:302] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters:
I20240503 08:12:58.022028 7964 raft_consensus.cc:2781] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:58.022809 7964 raft_consensus.cc:686] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:58.023284 7964 consensus_queue.cc:237] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:57.985199 7941 leader_election.cc:288] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Requested vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:12:58.015591 4770 leader_election.cc:302] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:58.032100 8065 raft_consensus.cc:2781] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:58.032342 8065 raft_consensus.cc:686] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:12:58.032758 8065 consensus_queue.cc:237] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:58.036470 8247 raft_consensus.cc:2932] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:58.055611 4565 raft_consensus.cc:2932] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:58.063030 4834 raft_consensus.cc:2932] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:58.077347 2203 raft_consensus.cc:2932] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:58.101032 1669 catalog_manager.cc:5484] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:58.119027 4963 raft_consensus.cc:2454] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:12:58.138116 8408 tablet_bootstrap.cc:492] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:58.138481 8408 ts_tablet_manager.cc:1367] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.366s user 0.001s sys 0.004s
I20240503 08:12:58.139657 8408 raft_consensus.cc:348] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:58.140089 8408 raft_consensus.cc:374] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.140228 8408 raft_consensus.cc:729] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.140697 8408 consensus_queue.cc:260] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:58.141968 8408 ts_tablet_manager.cc:1398] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:12:58.173594 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541925a798914d578021ce1f0d166730" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
W20240503 08:12:58.176254 1879 leader_election.cc:341] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:58.195461 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541925a798914d578021ce1f0d166730" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:58.206140 1670 catalog_manager.cc:5484] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:58.221866 1669 catalog_manager.cc:5484] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } }
W20240503 08:12:58.239292 1877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.197:42465, user_credentials={real_user=slave}} blocked reactor thread for 179689us
I20240503 08:12:58.252997 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3e931766d44f0aaf902cf0e950f83b" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:58.253476 4834 raft_consensus.cc:2454] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
I20240503 08:12:58.260211 4963 consensus_queue.cc:237] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:12:58.264035 1878 leader_election.cc:302] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters: 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:12:58.266311 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b22a007a2540ea9364579ac680fcf4" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:58.267938 1810 raft_consensus.cc:3037] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.270752 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b22a007a2540ea9364579ac680fcf4" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:58.284278 2201 raft_consensus.cc:1261] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:58.285091 8441 consensus_queue.cc:1035] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:58.285219 8360 raft_consensus.cc:2781] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:58.285403 8360 raft_consensus.cc:686] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:58.285663 8360 consensus_queue.cc:237] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:58.287961 8300 raft_consensus.cc:482] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:58.288197 8300 raft_consensus.cc:504] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:58.289265 8300 leader_election.cc:288] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:58.290223 8441 raft_consensus.cc:2932] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:58.296011 2204 raft_consensus.cc:2932] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:58.298796 1810 raft_consensus.cc:2454] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:12:58.303925 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09a268c672ba4580a2d93b5d15d12666 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:58.306344 1667 catalog_manager.cc:5484] T 09a268c672ba4580a2d93b5d15d12666 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:58.307364 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3e931766d44f0aaf902cf0e950f83b" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:58.307632 4700 raft_consensus.cc:2454] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:58.313370 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3e931766d44f0aaf902cf0e950f83b" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:12:58.313634 1811 raft_consensus.cc:2454] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 0.
W20240503 08:12:58.315101 4504 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.199:39455, user_credentials={real_user=slave}} blocked reactor thread for 166415us
I20240503 08:12:58.320456 8451 raft_consensus.cc:1050] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:12:58.321350 8451 consensus_queue.cc:237] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:12:58.328599 4565 raft_consensus.cc:1261] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:58.332285 1749 leader_election.cc:302] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:12:58.333565 4637 leader_election.cc:302] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters:
W20240503 08:12:58.334111 1879 leader_election.cc:341] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:12:58.334926 8111 raft_consensus.cc:2781] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:58.335040 8111 raft_consensus.cc:482] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:58.335144 8111 raft_consensus.cc:3037] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.337996 8111 raft_consensus.cc:504] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:58.338227 8441 consensus_queue.cc:1035] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:58.339180 8111 leader_election.cc:288] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:58.340500 4700 raft_consensus.cc:1261] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:58.341253 7742 consensus_queue.cc:1035] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:58.346761 4568 raft_consensus.cc:2932] T 09a268c672ba4580a2d93b5d15d12666 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:12:58.357162 8446 raft_consensus.cc:2781] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:58.357918 8401 tablet_bootstrap.cc:492] T 541925a798914d578021ce1f0d166730 P b8f581390da449ccb9e299011296405e: No bootstrap required, opened a new log
I20240503 08:12:58.358233 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3e931766d44f0aaf902cf0e950f83b" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:58.362143 8446 raft_consensus.cc:482] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:58.362406 8446 raft_consensus.cc:3037] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.368781 8446 raft_consensus.cc:504] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:58.369383 1811 raft_consensus.cc:2379] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 31f41542452b4b1eb994ff4223d039fc in current term 1: Already voted for candidate b8f581390da449ccb9e299011296405e in this term.
I20240503 08:12:58.370271 8446 leader_election.cc:288] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:58.371662 8401 ts_tablet_manager.cc:1367] T 541925a798914d578021ce1f0d166730 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.541s user 0.125s sys 0.154s
I20240503 08:12:58.373304 8401 raft_consensus.cc:348] T 541925a798914d578021ce1f0d166730 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:58.374783 8401 raft_consensus.cc:374] T 541925a798914d578021ce1f0d166730 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.375119 8401 raft_consensus.cc:729] T 541925a798914d578021ce1f0d166730 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.375648 8401 consensus_queue.cc:260] T 541925a798914d578021ce1f0d166730 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:58.376950 8401 ts_tablet_manager.cc:1398] T 541925a798914d578021ce1f0d166730 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:12:58.387018 1667 catalog_manager.cc:5484] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:58.388063 1746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.197:42465, user_credentials={real_user=slave}} blocked reactor thread for 50730.6us
I20240503 08:12:58.391546 4813 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a268c672ba4580a2d93b5d15d12666 with delete_type TABLET_DATA_TOMBSTONED (TS f24648c92c384f1ba349eee801f985e2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33952
I20240503 08:12:58.393956 2072 raft_consensus.cc:1261] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:58.395893 8460 tablet_replica.cc:330] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:12:58.396198 8460 raft_consensus.cc:2229] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:58.396600 8460 raft_consensus.cc:2258] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:58.405179 8460 ts_tablet_manager.cc:1875] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:58.408015 1807 raft_consensus.cc:3037] T 541925a798914d578021ce1f0d166730 P b8f581390da449ccb9e299011296405e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.412147 7964 consensus_queue.cc:1035] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:12:58.414944 1747 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.195:43117, user_credentials={real_user=slave}} blocked reactor thread for 52411us
I20240503 08:12:58.419375 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3e931766d44f0aaf902cf0e950f83b" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:58.419869 4834 raft_consensus.cc:2454] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 0.
I20240503 08:12:58.422869 8460 ts_tablet_manager.cc:1888] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:58.425838 8460 log.cc:1196] T 09a268c672ba4580a2d93b5d15d12666 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/09a268c672ba4580a2d93b5d15d12666
I20240503 08:12:58.430757 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3e931766d44f0aaf902cf0e950f83b" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:58.431057 4834 raft_consensus.cc:3037] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.431092 1812 raft_consensus.cc:1261] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:58.438077 4834 raft_consensus.cc:2454] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
W20240503 08:12:58.440726 1749 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.198:46559, user_credentials={real_user=slave}} blocked reactor thread for 62811.2us
W20240503 08:12:58.441524 8417 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.038s sys 0.056s
I20240503 08:12:58.441555 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3e931766d44f0aaf902cf0e950f83b" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
W20240503 08:12:58.441823 8417 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.664s user 0.038s sys 0.056s
I20240503 08:12:58.441865 4699 raft_consensus.cc:2379] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8f581390da449ccb9e299011296405e in current term 1: Already voted for candidate 31f41542452b4b1eb994ff4223d039fc in this term.
I20240503 08:12:58.466293 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet 09a268c672ba4580a2d93b5d15d12666 (table rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]) successfully deleted
I20240503 08:12:58.473448 8417 tablet_bootstrap.cc:492] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:58.473677 8417 ts_tablet_manager.cc:1367] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.700s user 0.041s sys 0.056s
I20240503 08:12:58.473915 8443 consensus_queue.cc:1035] T 541925a798914d578021ce1f0d166730 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.445910 8431 consensus_queue.cc:1035] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:58.452415 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3e931766d44f0aaf902cf0e950f83b" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:58.487813 8161 consensus_queue.cc:1035] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:58.464358 7742 raft_consensus.cc:1050] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a to VOTER
I20240503 08:12:58.489382 7742 consensus_queue.cc:237] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:12:58.493477 4568 raft_consensus.cc:1261] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:58.494172 4699 raft_consensus.cc:1261] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:58.495312 8432 consensus_queue.cc:1035] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:58.495517 8451 raft_consensus.cc:2932] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:58.496191 8432 consensus_queue.cc:1035] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:58.498721 1942 raft_consensus.cc:1261] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:12:58.499925 1810 raft_consensus.cc:2932] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:58.504004 1670 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 08:12:58.503944 7742 consensus_queue.cc:1035] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:12:58.507489 4638 leader_election.cc:302] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters: b8f581390da449ccb9e299011296405e
I20240503 08:12:58.508078 8439 raft_consensus.cc:2781] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:58.508285 8439 raft_consensus.cc:686] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:12:58.508630 8439 consensus_queue.cc:237] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:58.521498 4700 raft_consensus.cc:2932] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:58.518843 1669 catalog_manager.cc:5484] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:12:58.546236 4835 consensus_queue.cc:237] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:12:58.546458 1749 leader_election.cc:302] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f581390da449ccb9e299011296405e; no voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2
I20240503 08:12:58.546886 8458 raft_consensus.cc:2726] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 08:12:58.553788 4700 raft_consensus.cc:1261] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:58.554556 8451 consensus_queue.cc:1035] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:58.557884 7742 raft_consensus.cc:2932] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:58.560884 1807 raft_consensus.cc:1261] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:58.561944 8161 consensus_queue.cc:1035] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:58.563733 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 929d8d354b914502b5e677ccc84e2eda with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:58.566542 1669 catalog_manager.cc:5484] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:58.567808 8065 raft_consensus.cc:2932] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:58.568014 4699 raft_consensus.cc:2932] T 929d8d354b914502b5e677ccc84e2eda P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:58.572947 4698 raft_consensus.cc:2932] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:58.573668 4568 raft_consensus.cc:2932] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:58.575513 8451 consensus_queue.cc:1035] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.595602 8161 consensus_queue.cc:1035] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.595960 1667 catalog_manager.cc:5484] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:58.612811 4965 raft_consensus.cc:1261] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:58.622069 2052 tablet_service.cc:1514] Processing DeleteTablet for tablet 929d8d354b914502b5e677ccc84e2eda with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:58.651222 8443 consensus_queue.cc:1035] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.652295 1812 raft_consensus.cc:2932] T 929d8d354b914502b5e677ccc84e2eda P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:12:58.666625 8417 raft_consensus.cc:348] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:58.666921 8417 raft_consensus.cc:374] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.667037 8417 raft_consensus.cc:729] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.667366 8417 consensus_queue.cc:260] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } }
I20240503 08:12:58.675204 8417 ts_tablet_manager.cc:1398] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.192s user 0.002s sys 0.000s
I20240503 08:12:58.676746 2071 raft_consensus.cc:3037] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.677222 1667 catalog_manager.cc:5576] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117) does not have the latest schema for tablet 58b502e473224315806bf90ab972eb82 (table rebalancer_dynamic_table [id=9aceb057f2af4ae2a244a43374d2a9a1]). Expected version 1 got 0
I20240503 08:12:58.685226 2071 raft_consensus.cc:1261] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:58.688925 1667 catalog_manager.cc:5576] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117) does not have the latest schema for tablet 58b502e473224315806bf90ab972eb82 (table rebalancer_dynamic_table [id=9aceb057f2af4ae2a244a43374d2a9a1]). Expected version 1 got 0
I20240503 08:12:58.696820 8485 tablet_replica.cc:330] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:58.697196 8485 raft_consensus.cc:2229] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:58.697521 8485 pending_rounds.cc:62] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a: Trying to abort 1 pending ops.
I20240503 08:12:58.697696 8485 pending_rounds.cc:68] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023509415201943552 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "929d8d354b914502b5e677ccc84e2eda" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } } }
I20240503 08:12:58.698441 8485 raft_consensus.cc:2894] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 08:12:58.698669 8485 raft_consensus.cc:2866] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:12:58.698860 8485 raft_consensus.cc:2258] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:58.699235 1668 catalog_manager.cc:5484] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:58.733790 1878 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.200:45283, user_credentials={real_user=slave}} blocked reactor thread for 82770.8us
I20240503 08:12:58.764205 4835 consensus_queue.cc:237] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:12:58.767585 4698 raft_consensus.cc:1261] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:58.768582 8451 consensus_queue.cc:1035] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:58.773926 8451 raft_consensus.cc:2932] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:58.787950 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98ad826dca7c4343afcae34bf1a40da8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:58.789399 1668 catalog_manager.cc:5484] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:58.794287 4547 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ad826dca7c4343afcae34bf1a40da8 with delete_type TABLET_DATA_TOMBSTONED (TS 5f95b05d111e4090bbc59dec394ed0ab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39444
I20240503 08:12:58.803225 4698 raft_consensus.cc:2932] T 98ad826dca7c4343afcae34bf1a40da8 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:58.816910 8490 tablet_replica.cc:330] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:12:58.817457 8490 raft_consensus.cc:2229] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:58.817864 8490 raft_consensus.cc:2258] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:58.819001 8490 ts_tablet_manager.cc:1875] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:58.846441 8485 ts_tablet_manager.cc:1875] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:58.847167 7964 consensus_queue.cc:1035] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.864358 1942 raft_consensus.cc:2932] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:58.869122 8490 ts_tablet_manager.cc:1888] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:58.869374 8490 log.cc:1196] T 98ad826dca7c4343afcae34bf1a40da8 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/98ad826dca7c4343afcae34bf1a40da8
I20240503 08:12:58.870879 1942 raft_consensus.cc:1261] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:12:58.877573 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 98ad826dca7c4343afcae34bf1a40da8 (table rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]) successfully deleted
I20240503 08:12:58.882469 1667 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 58b502e473224315806bf90ab972eb82 (table rebalancer_dynamic_table [id=9aceb057f2af4ae2a244a43374d2a9a1]). Expected version 1 got 0
I20240503 08:12:58.902088 1942 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1cf9b2eef42e4fff8cdf5f9dd7b8b31e"
dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47580
I20240503 08:12:58.902340 1942 raft_consensus.cc:595] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Received request to transfer leadership
I20240503 08:12:58.909817 7742 consensus_queue.cc:1035] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:12:58.915591 8443 raft_consensus.cc:979] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: : Instructing follower 00b61625d8a7460892c118a2570afda5 to start an election
I20240503 08:12:58.915794 8443 raft_consensus.cc:1067] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Signalling peer 00b61625d8a7460892c118a2570afda5 to start an election
I20240503 08:12:58.934402 8497 tablet.cc:1722] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:58.935117 8497 tablet.cc:1620] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: MemRowSet was empty: no flush needed.
I20240503 08:12:58.945791 1667 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=9aceb057f2af4ae2a244a43374d2a9a1] alter complete (version 1)
I20240503 08:12:58.947094 4965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1cf9b2eef42e4fff8cdf5f9dd7b8b31e"
dest_uuid: "00b61625d8a7460892c118a2570afda5"
from {username='slave'} at 127.27.157.194:41645
I20240503 08:12:58.947371 4965 raft_consensus.cc:482] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:58.947561 4965 raft_consensus.cc:3037] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:58.950227 1942 raft_consensus.cc:2932] T 98ad826dca7c4343afcae34bf1a40da8 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:12:58.951845 4965 raft_consensus.cc:504] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:58.961606 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9b2eef42e4fff8cdf5f9dd7b8b31e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:12:58.961853 1812 raft_consensus.cc:3037] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:58.965152 1812 raft_consensus.cc:2454] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 2.
I20240503 08:12:58.967959 8485 ts_tablet_manager.cc:1888] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:12:58.968087 8314 consensus_queue.cc:1035] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.975033 8485 log.cc:1196] T 929d8d354b914502b5e677ccc84e2eda P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/929d8d354b914502b5e677ccc84e2eda
I20240503 08:12:58.976110 1667 catalog_manager.cc:5576] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117) does not have the latest schema for tablet 58b502e473224315806bf90ab972eb82 (table rebalancer_dynamic_table [id=9aceb057f2af4ae2a244a43374d2a9a1]). Expected version 1 got 0
I20240503 08:12:58.976406 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 929d8d354b914502b5e677ccc84e2eda (table rebalancer_extra_table_18 [id=8a48fca96c0e4fdfa7cd9f966b583e55]) successfully deleted
I20240503 08:12:58.983230 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9b2eef42e4fff8cdf5f9dd7b8b31e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:58.983546 1942 raft_consensus.cc:3032] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:58.983623 1942 raft_consensus.cc:729] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:12:58.983903 1942 consensus_queue.cc:260] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:58.984364 1942 raft_consensus.cc:3037] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:58.986946 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9b2eef42e4fff8cdf5f9dd7b8b31e" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:58.987257 4835 raft_consensus.cc:3037] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:58.990947 4835 raft_consensus.cc:2454] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 2.
I20240503 08:12:58.991581 4903 leader_election.cc:302] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, b8f581390da449ccb9e299011296405e, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:58.992329 8440 raft_consensus.cc:2781] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:58.992491 8440 raft_consensus.cc:686] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:58.992833 8440 consensus_queue.cc:237] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:58.995874 1942 raft_consensus.cc:2454] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 2.
I20240503 08:12:59.001376 8502 tablet.cc:1722] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:59.002966 8439 consensus_queue.cc:1035] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.003803 4965 leader_election.cc:288] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:59.005348 1668 catalog_manager.cc:5576] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117) does not have the latest schema for tablet 58b502e473224315806bf90ab972eb82 (table rebalancer_dynamic_table [id=9aceb057f2af4ae2a244a43374d2a9a1]). Expected version 1 got 0
I20240503 08:12:59.013995 8502 tablet.cc:1620] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: MemRowSet was empty: no flush needed.
I20240503 08:12:59.042905 8499 tablet.cc:1722] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 08:12:59.044247 8499 tablet.cc:1620] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:59.059202 1669 catalog_manager.cc:5576] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283) does not have the latest schema for tablet 58b502e473224315806bf90ab972eb82 (table rebalancer_dynamic_table [id=9aceb057f2af4ae2a244a43374d2a9a1]). Expected version 1 got 0
I20240503 08:12:59.064566 1669 catalog_manager.cc:5484] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 1 to 2, leader changed from 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:59.086987 1669 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 08:12:59.102880 8497 tablet.cc:1722] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:59.103435 8497 tablet.cc:1620] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: MemRowSet was empty: no flush needed.
I20240503 08:12:59.105355 8499 tablet.cc:1722] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:59.105875 8499 tablet.cc:1620] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: MemRowSet was empty: no flush needed.
I20240503 08:12:59.106905 8502 tablet.cc:1722] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 08:12:59.107432 8502 tablet.cc:1620] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: MemRowSet was empty: no flush needed.
I20240503 08:12:59.132615 1670 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table [id=9aceb057f2af4ae2a244a43374d2a9a1] alter complete (version 2)
I20240503 08:12:59.238402 1670 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 08:12:59.251073 8497 tablet.cc:1722] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:59.260280 8502 tablet.cc:1722] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:59.270148 8499 tablet.cc:1722] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 08:12:59.273727 1668 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: rebalancer_dynamic_table_new_name [id=9aceb057f2af4ae2a244a43374d2a9a1] alter complete (version 3)
I20240503 08:12:59.340812 1668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:12:59.347916 4831 consensus_queue.cc:237] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:59.350841 2052 tablet_service.cc:1467] Processing CreateTablet for tablet c2d33b2dc884448483072fa6ab825f0b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ba3321da10e94c50a92ea8114756e5ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.351410 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d33b2dc884448483072fa6ab825f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.351840 4547 tablet_service.cc:1467] Processing CreateTablet for tablet c2d33b2dc884448483072fa6ab825f0b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ba3321da10e94c50a92ea8114756e5ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.352298 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d33b2dc884448483072fa6ab825f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.352388 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 4fba9f969e764737973456dbdb650b1d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ba3321da10e94c50a92ea8114756e5ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.352537 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae7520f758c47a39cfa3aaf59ebc674 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ba3321da10e94c50a92ea8114756e5ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.352849 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fba9f969e764737973456dbdb650b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.352905 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae7520f758c47a39cfa3aaf59ebc674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.358400 4813 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae7520f758c47a39cfa3aaf59ebc674 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ba3321da10e94c50a92ea8114756e5ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.358779 4683 tablet_service.cc:1467] Processing CreateTablet for tablet c2d33b2dc884448483072fa6ab825f0b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ba3321da10e94c50a92ea8114756e5ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.358924 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae7520f758c47a39cfa3aaf59ebc674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.359241 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d33b2dc884448483072fa6ab825f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.366303 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae7520f758c47a39cfa3aaf59ebc674 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ba3321da10e94c50a92ea8114756e5ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.366984 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae7520f758c47a39cfa3aaf59ebc674. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.373154 8570 tablet_bootstrap.cc:492] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:59.374102 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 4fba9f969e764737973456dbdb650b1d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ba3321da10e94c50a92ea8114756e5ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:59.374673 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fba9f969e764737973456dbdb650b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.379856 8572 tablet_bootstrap.cc:492] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:59.381882 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 4fba9f969e764737973456dbdb650b1d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ba3321da10e94c50a92ea8114756e5ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
W20240503 08:12:59.382160 4769 consensus_peers.cc:485] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4a205bf277d4770836dbbf0ec28d884. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:59.382388 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fba9f969e764737973456dbdb650b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.382601 8572 tablet_bootstrap.cc:654] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.383255 4967 raft_consensus.cc:1261] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:59.384248 4698 raft_consensus.cc:1261] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:59.384296 8451 consensus_queue.cc:1035] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:59.384943 8451 consensus_queue.cc:1035] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:59.393610 8572 tablet_bootstrap.cc:492] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:59.393872 8572 ts_tablet_manager.cc:1367] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20240503 08:12:59.395700 1668 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:59.395959 1668 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 08:12:59.396006 8572 raft_consensus.cc:348] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:59.396427 8572 raft_consensus.cc:374] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.396592 8572 raft_consensus.cc:729] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.396970 8572 consensus_queue.cc:260] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:59.399506 8572 ts_tablet_manager.cc:1398] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:12:59.399991 8572 tablet_bootstrap.cc:492] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:59.402280 8572 tablet_bootstrap.cc:654] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.404414 1668 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f97fddd2586f4c5cb4a95a0cbf96c0d0: Sending DeleteTablet for 3 replicas of tablet 58b502e473224315806bf90ab972eb82
I20240503 08:12:59.404035 8161 raft_consensus.cc:2932] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:59.405776 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 58b502e473224315806bf90ab972eb82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:59 UTC) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:59.406193 4944 tablet_service.cc:1514] Processing DeleteTablet for tablet 58b502e473224315806bf90ab972eb82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:59 UTC) from {username='slave'} at 127.0.0.1:47168
I20240503 08:12:59.414968 4703 raft_consensus.cc:2932] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:59.423673 1670 catalog_manager.cc:5484] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:59.434975 8581 tablet_replica.cc:330] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:12:59.435523 8581 raft_consensus.cc:2229] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:59.435915 8581 raft_consensus.cc:2258] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.439785 8578 tablet_bootstrap.cc:492] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:59.437552 8581 ts_tablet_manager.cc:1875] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:59.441468 8580 tablet_replica.cc:330] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:59.441790 8580 raft_consensus.cc:2229] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:59.442243 8580 raft_consensus.cc:2258] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.443527 8572 tablet_bootstrap.cc:492] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:59.443560 8580 ts_tablet_manager.cc:1875] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:59.443774 8572 ts_tablet_manager.cc:1367] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.008s
I20240503 08:12:59.447806 8571 tablet_bootstrap.cc:492] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:12:59.448395 8578 tablet_bootstrap.cc:654] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.449519 8572 raft_consensus.cc:348] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:59.451468 8572 raft_consensus.cc:374] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.452159 8576 tablet_bootstrap.cc:492] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:12:59.452282 8572 raft_consensus.cc:729] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.454056 8576 tablet_bootstrap.cc:654] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.454941 8572 consensus_queue.cc:260] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:59.417685 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 58b502e473224315806bf90ab972eb82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 08:12:59 UTC) from {username='slave'} at 127.0.0.1:35210
I20240503 08:12:59.455931 8485 tablet_replica.cc:330] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:12:59.455962 8571 tablet_bootstrap.cc:654] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.456171 8485 raft_consensus.cc:2229] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:59.456216 8572 ts_tablet_manager.cc:1398] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240503 08:12:59.456404 8485 raft_consensus.cc:2258] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.457911 8570 tablet_bootstrap.cc:654] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.464102 8485 ts_tablet_manager.cc:1875] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 08:12:59.464042 4963 raft_consensus.cc:2932] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:59.469398 8581 ts_tablet_manager.cc:1888] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:59.471637 8578 tablet_bootstrap.cc:492] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:59.471951 8578 ts_tablet_manager.cc:1367] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.004s
I20240503 08:12:59.473011 8578 raft_consensus.cc:348] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:59.473480 8581 log.cc:1196] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/58b502e473224315806bf90ab972eb82
I20240503 08:12:59.474146 8581 ts_tablet_manager.cc:1909] T 58b502e473224315806bf90ab972eb82 P 00b61625d8a7460892c118a2570afda5: Deleting consensus metadata
I20240503 08:12:59.474334 8578 raft_consensus.cc:374] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.474531 8578 raft_consensus.cc:729] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.475184 8578 consensus_queue.cc:260] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:59.480763 8576 tablet_bootstrap.cc:492] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5: No bootstrap required, opened a new log
I20240503 08:12:59.481737 8576 ts_tablet_manager.cc:1367] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240503 08:12:59.482334 8570 tablet_bootstrap.cc:492] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:59.482574 8570 ts_tablet_manager.cc:1367] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.000s
I20240503 08:12:59.482647 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 58b502e473224315806bf90ab972eb82 (table rebalancer_dynamic_table_new_name [id=9aceb057f2af4ae2a244a43374d2a9a1]) successfully deleted
I20240503 08:12:59.483469 8485 ts_tablet_manager.cc:1888] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:59.483744 8485 log.cc:1196] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/58b502e473224315806bf90ab972eb82
I20240503 08:12:59.483774 8578 ts_tablet_manager.cc:1398] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240503 08:12:59.483624 8570 raft_consensus.cc:348] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:59.484520 8577 tablet_bootstrap.cc:492] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:59.484612 8570 raft_consensus.cc:374] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.484791 8570 raft_consensus.cc:729] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.485674 8578 tablet_bootstrap.cc:492] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:12:59.483464 8576 raft_consensus.cc:348] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:59.486968 8576 raft_consensus.cc:374] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.487144 8576 raft_consensus.cc:729] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.485425 8570 consensus_queue.cc:260] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:59.488634 8576 consensus_queue.cc:260] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:59.489336 8576 ts_tablet_manager.cc:1398] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:12:59.489568 8578 tablet_bootstrap.cc:654] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.493635 8571 tablet_bootstrap.cc:492] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5: No bootstrap required, opened a new log
I20240503 08:12:59.493815 8571 ts_tablet_manager.cc:1367] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.000s
I20240503 08:12:59.494879 8570 ts_tablet_manager.cc:1398] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 08:12:59.495234 8570 tablet_bootstrap.cc:492] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:12:59.495131 8571 raft_consensus.cc:348] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:59.495412 8571 raft_consensus.cc:374] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.495551 8571 raft_consensus.cc:729] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.496147 8571 consensus_queue.cc:260] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:59.497509 8570 tablet_bootstrap.cc:654] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.498090 8485 ts_tablet_manager.cc:1909] T 58b502e473224315806bf90ab972eb82 P 43e2e01cdd314f9c872c4336ef74250a: Deleting consensus metadata
I20240503 08:12:59.500648 8571 ts_tablet_manager.cc:1398] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240503 08:12:59.505468 8580 ts_tablet_manager.cc:1888] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 08:12:59.506112 8577 tablet_bootstrap.cc:654] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.506597 8580 log.cc:1196] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/58b502e473224315806bf90ab972eb82
I20240503 08:12:59.507663 8580 ts_tablet_manager.cc:1909] T 58b502e473224315806bf90ab972eb82 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting consensus metadata
I20240503 08:12:59.508076 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 58b502e473224315806bf90ab972eb82 (table rebalancer_dynamic_table_new_name [id=9aceb057f2af4ae2a244a43374d2a9a1]) successfully deleted
I20240503 08:12:59.508394 1667 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 08:12:59.509265 8578 tablet_bootstrap.cc:492] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2: No bootstrap required, opened a new log
I20240503 08:12:59.510947 8578 ts_tablet_manager.cc:1367] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.002s
I20240503 08:12:59.512231 8570 tablet_bootstrap.cc:492] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a: No bootstrap required, opened a new log
I20240503 08:12:59.512401 8570 ts_tablet_manager.cc:1367] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 08:12:59.512358 8578 raft_consensus.cc:348] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:59.513218 8570 raft_consensus.cc:348] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:59.513649 8570 raft_consensus.cc:374] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.513831 8570 raft_consensus.cc:729] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.514183 8570 consensus_queue.cc:260] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:59.512691 8578 raft_consensus.cc:374] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.516501 8570 ts_tablet_manager.cc:1398] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:12:59.516515 8578 raft_consensus.cc:729] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.516870 8578 consensus_queue.cc:260] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:59.519150 8578 ts_tablet_manager.cc:1398] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 08:12:59.522953 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa23eb4acce4d6f8733eb80689c57e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab10b035aeca441487649ab01e447cb7]), partition=
I20240503 08:12:59.523460 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa23eb4acce4d6f8733eb80689c57e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.525905 4548 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa23eb4acce4d6f8733eb80689c57e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab10b035aeca441487649ab01e447cb7]), partition=
I20240503 08:12:59.526361 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa23eb4acce4d6f8733eb80689c57e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.531991 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 58b502e473224315806bf90ab972eb82 (table rebalancer_dynamic_table_new_name [id=9aceb057f2af4ae2a244a43374d2a9a1]) successfully deleted
I20240503 08:12:59.532169 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa23eb4acce4d6f8733eb80689c57e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ab10b035aeca441487649ab01e447cb7]), partition=
I20240503 08:12:59.532574 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa23eb4acce4d6f8733eb80689c57e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:59.536065 4831 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:12:59.537616 8440 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:12:59.538205 8572 tablet_bootstrap.cc:492] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:12:59.540246 8572 tablet_bootstrap.cc:654] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.553881 4831 consensus_queue.cc:237] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:12:59.555461 8577 tablet_bootstrap.cc:492] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:59.555745 8577 ts_tablet_manager.cc:1367] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.003s
I20240503 08:12:59.556957 8577 raft_consensus.cc:348] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:59.557302 8577 raft_consensus.cc:374] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.557469 8577 raft_consensus.cc:729] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.558450 8577 consensus_queue.cc:260] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:59.561445 8572 tablet_bootstrap.cc:492] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab: No bootstrap required, opened a new log
I20240503 08:12:59.563704 8577 ts_tablet_manager.cc:1398] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:12:59.564121 8577 tablet_bootstrap.cc:492] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:12:59.565049 8572 ts_tablet_manager.cc:1367] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.006s
I20240503 08:12:59.566018 8577 tablet_bootstrap.cc:654] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.568379 8572 raft_consensus.cc:348] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:59.573696 8572 raft_consensus.cc:374] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.573966 8572 raft_consensus.cc:729] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.574014 1812 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:12:59.574420 8572 consensus_queue.cc:260] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:59.575241 1943 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:12:59.576054 8572 ts_tablet_manager.cc:1398] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240503 08:12:59.575932 8593 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:12:59.576486 8593 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:12:59.577726 8597 tablet_bootstrap.cc:492] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:12:59.579378 8597 tablet_bootstrap.cc:654] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:59.582120 8577 tablet_bootstrap.cc:492] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc: No bootstrap required, opened a new log
I20240503 08:12:59.582398 8577 ts_tablet_manager.cc:1367] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.006s
I20240503 08:12:59.583956 4703 raft_consensus.cc:1261] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:59.584597 4565 raft_consensus.cc:1261] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:59.585165 8161 consensus_queue.cc:1035] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 08:12:59.584160 8577 raft_consensus.cc:348] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:59.586055 8451 consensus_queue.cc:1035] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:59.586230 8577 raft_consensus.cc:374] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.586349 8577 raft_consensus.cc:729] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.586652 8577 consensus_queue.cc:260] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:59.589125 8577 ts_tablet_manager.cc:1398] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20240503 08:12:59.591923 4768 consensus_peers.cc:485] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df0c416025c74075b6429d169fcdd58c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:59.612012 8597 tablet_bootstrap.cc:492] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: No bootstrap required, opened a new log
I20240503 08:12:59.612332 8597 ts_tablet_manager.cc:1367] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.009s
I20240503 08:12:59.607163 7742 raft_consensus.cc:2932] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:59.613271 8597 raft_consensus.cc:348] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:59.613601 8597 raft_consensus.cc:374] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:59.613703 8597 raft_consensus.cc:729] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: FOLLOWER
I20240503 08:12:59.614010 8597 consensus_queue.cc:260] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:59.614776 8597 ts_tablet_manager.cc:1398] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:12:59.616031 8560 raft_consensus.cc:482] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.616209 8560 raft_consensus.cc:504] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:59.616466 4698 raft_consensus.cc:2932] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:59.617061 8560 leader_election.cc:288] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:59.617812 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae7520f758c47a39cfa3aaf59ebc674" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:12:59.618049 2071 raft_consensus.cc:2454] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:59.618645 4502 leader_election.cc:302] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:59.619290 8448 raft_consensus.cc:2781] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.619415 8448 raft_consensus.cc:482] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.619531 8448 raft_consensus.cc:3037] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.619824 4565 raft_consensus.cc:2932] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:59.622107 8448 raft_consensus.cc:504] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:59.624337 8448 leader_election.cc:288] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:12:59.626998 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae7520f758c47a39cfa3aaf59ebc674" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:12:59.627260 2071 raft_consensus.cc:3037] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.627709 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae7520f758c47a39cfa3aaf59ebc674" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:59.627929 4835 raft_consensus.cc:3037] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.628805 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae7520f758c47a39cfa3aaf59ebc674" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:59.630247 2071 raft_consensus.cc:2454] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:59.630874 4835 raft_consensus.cc:2454] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:59.633100 4967 consensus_queue.cc:237] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:12:59.635478 1812 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:12:59.636111 8440 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:12:59.637046 4504 leader_election.cc:302] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:59.637236 1670 catalog_manager.cc:5484] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:12:59.638152 8560 raft_consensus.cc:2781] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.638322 8560 raft_consensus.cc:686] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:59.638608 8560 consensus_queue.cc:237] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:12:59.638792 8440 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:59.643265 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:12:59.646035 1812 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:59.647420 1667 catalog_manager.cc:5484] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 8 to 10, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:59.657265 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50100
I20240503 08:12:59.657835 8580 tablet_replica.cc:330] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:12:59.658157 8580 raft_consensus.cc:2229] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:59.658376 8580 raft_consensus.cc:2258] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:59.667631 4834 raft_consensus.cc:1261] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:12:59.668586 8593 consensus_queue.cc:1035] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:12:59.669020 1668 catalog_manager.cc:5484] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:59.670280 4834 raft_consensus.cc:2932] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:12:59.681972 8340 raft_consensus.cc:482] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.682188 8340 raft_consensus.cc:504] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:59.683085 8340 leader_election.cc:288] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:59.684321 4565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33b2dc884448483072fa6ab825f0b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab" is_pre_election: true
I20240503 08:12:59.684650 4565 raft_consensus.cc:2454] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:59.685648 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33b2dc884448483072fa6ab825f0b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:59.685914 4698 raft_consensus.cc:2454] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 0.
I20240503 08:12:59.686575 8580 ts_tablet_manager.cc:1875] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:12:59.687183 2007 leader_election.cc:302] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:59.689540 8340 raft_consensus.cc:2781] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.689667 8340 raft_consensus.cc:482] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.689764 8340 raft_consensus.cc:3037] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.696144 8340 raft_consensus.cc:504] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:59.697578 8580 ts_tablet_manager.cc:1888] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:12:59.697738 8580 log.cc:1196] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/1cf9b2eef42e4fff8cdf5f9dd7b8b31e
I20240503 08:12:59.697737 4565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33b2dc884448483072fa6ab825f0b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f95b05d111e4090bbc59dec394ed0ab"
I20240503 08:12:59.698100 4565 raft_consensus.cc:3037] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.698712 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e (table rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]) successfully deleted
I20240503 08:12:59.705924 4565 raft_consensus.cc:2454] T c2d33b2dc884448483072fa6ab825f0b P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:59.707091 2007 leader_election.cc:302] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:59.707592 8486 raft_consensus.cc:2781] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.708608 8607 raft_consensus.cc:482] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.708779 8607 raft_consensus.cc:504] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:59.709669 8607 leader_election.cc:288] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:59.710485 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa23eb4acce4d6f8733eb80689c57e6" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:12:59.710773 1942 raft_consensus.cc:2454] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:59.710719 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa23eb4acce4d6f8733eb80689c57e6" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:12:59.710983 4698 raft_consensus.cc:2454] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 0.
I20240503 08:12:59.711280 4503 leader_election.cc:302] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:59.711661 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33b2dc884448483072fa6ab825f0b" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:59.711774 8607 raft_consensus.cc:2781] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.711884 4698 raft_consensus.cc:3037] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.711884 8607 raft_consensus.cc:482] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.711978 8607 raft_consensus.cc:3037] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.715072 4698 raft_consensus.cc:2454] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:12:59.715741 8340 leader_election.cc:288] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 1 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:12:59.717343 8607 raft_consensus.cc:504] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:59.718559 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa23eb4acce4d6f8733eb80689c57e6" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:12:59.718822 4698 raft_consensus.cc:3037] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.719988 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa23eb4acce4d6f8733eb80689c57e6" candidate_uuid: "5f95b05d111e4090bbc59dec394ed0ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:12:59.720196 1942 raft_consensus.cc:3037] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.722169 4698 raft_consensus.cc:2454] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:59.723063 1942 raft_consensus.cc:2454] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f95b05d111e4090bbc59dec394ed0ab in term 1.
I20240503 08:12:59.724264 4503 leader_election.cc:302] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 5f95b05d111e4090bbc59dec394ed0ab; no voters:
I20240503 08:12:59.724649 8440 raft_consensus.cc:482] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:59.724720 8560 raft_consensus.cc:2781] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.724820 8440 raft_consensus.cc:504] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:59.724895 8560 raft_consensus.cc:686] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Running, Role: LEADER
I20240503 08:12:59.725237 8560 consensus_queue.cc:237] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:12:59.725580 8440 leader_election.cc:288] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:59.726698 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fba9f969e764737973456dbdb650b1d" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:12:59.726723 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fba9f969e764737973456dbdb650b1d" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:12:59.726939 4831 raft_consensus.cc:2454] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:59.726933 2203 raft_consensus.cc:2454] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 0.
I20240503 08:12:59.727468 4903 leader_election.cc:302] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:12:59.727661 8607 leader_election.cc:288] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [CANDIDATE]: Term 1 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:12:59.728034 8440 raft_consensus.cc:2781] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:59.728152 8440 raft_consensus.cc:482] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:59.728235 8440 raft_consensus.cc:3037] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.736403 8440 raft_consensus.cc:504] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:59.737567 2203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fba9f969e764737973456dbdb650b1d" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:12:59.737802 2203 raft_consensus.cc:3037] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.740408 1668 catalog_manager.cc:5484] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197). New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:59.740756 2203 raft_consensus.cc:2454] T 4fba9f969e764737973456dbdb650b1d P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:59.741281 4901 leader_election.cc:302] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 184c543cb05d49cc98ba1c56617be7a5; no voters:
I20240503 08:12:59.741604 8593 raft_consensus.cc:2781] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:59.741783 8593 raft_consensus.cc:686] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:12:59.742125 8593 consensus_queue.cc:237] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:12:59.744554 8486 raft_consensus.cc:686] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:12:59.744719 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fba9f969e764737973456dbdb650b1d" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:12:59.744957 4831 raft_consensus.cc:3037] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:59.745265 8486 consensus_queue.cc:237] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:12:59.747954 4831 raft_consensus.cc:2454] T 4fba9f969e764737973456dbdb650b1d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:12:59.742127 8440 leader_election.cc:288] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 1 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:12:59.759923 1668 catalog_manager.cc:5484] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:59.766805 4698 consensus_queue.cc:237] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:12:59.770131 1669 catalog_manager.cc:5484] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 0 to 1, leader changed from <none> to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:59.779222 4636 consensus_peers.cc:485] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01821d71873d4ad78d7b7f685161a229. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:59.780359 2071 raft_consensus.cc:1261] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:59.781080 8314 consensus_queue.cc:1035] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:59.789299 4565 raft_consensus.cc:1261] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:59.790513 8439 consensus_queue.cc:1035] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:59.799525 8439 raft_consensus.cc:2932] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:59.800438 4568 raft_consensus.cc:2932] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:59.814967 1669 catalog_manager.cc:5484] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:59.836256 2072 raft_consensus.cc:2932] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:12:59.871954 2071 consensus_queue.cc:237] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
W20240503 08:12:59.875656 2009 consensus_peers.cc:485] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d3aaa1025ca4b0e832535d1ca3d5c4f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:59.875911 4963 raft_consensus.cc:1261] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:59.876474 8617 consensus_queue.cc:1035] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:59.876781 4831 raft_consensus.cc:1261] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:59.877449 8617 consensus_queue.cc:1035] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:59.884958 8486 raft_consensus.cc:2932] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:59.889509 4831 raft_consensus.cc:2932] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:59.892180 1669 catalog_manager.cc:5484] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:59.916321 4963 raft_consensus.cc:2932] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:12:59.928990 2203 consensus_queue.cc:237] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:12:59.944820 4703 raft_consensus.cc:1261] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:59.944820 1942 raft_consensus.cc:1261] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:12:59.944983 2139 consensus_peers.cc:485] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe20c189635648f5b78905e8b9643bcd. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:59.945626 7924 consensus_queue.cc:1035] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:59.945937 8027 consensus_queue.cc:1035] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:12:59.948661 7924 raft_consensus.cc:2932] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:59.949476 1942 raft_consensus.cc:2932] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:59.951887 4703 raft_consensus.cc:2932] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:59.956296 1668 catalog_manager.cc:5484] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:12:59.994359 8652 ts_tablet_manager.cc:902] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:13:00.002383 4963 consensus_queue.cc:237] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:00.008598 4831 raft_consensus.cc:1261] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:00.008829 4565 raft_consensus.cc:1261] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:00.009326 8247 consensus_queue.cc:1035] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:00.010046 4901 consensus_peers.cc:485] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6994370a532472893cc934337f3cb6e. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.010588 8247 consensus_queue.cc:1035] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:00.031149 8247 raft_consensus.cc:2932] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:00.031955 8652 tablet_copy_client.cc:308] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:13:00.032703 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet f4a205bf277d4770836dbbf0ec28d884 from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:54589)
I20240503 08:13:00.032857 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet f4a205bf277d4770836dbbf0ec28d884 from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:54589: session id = b8f581390da449ccb9e299011296405e-f4a205bf277d4770836dbbf0ec28d884
I20240503 08:13:00.035813 4855 tablet_copy_source_session.cc:205] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:00.035506 4831 raft_consensus.cc:2932] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:00.037292 8652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a205bf277d4770836dbbf0ec28d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.041363 4568 raft_consensus.cc:2932] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:00.044067 1668 catalog_manager.cc:5484] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:00.045259 8652 tablet_copy_client.cc:783] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:00.045522 8652 tablet_copy_client.cc:647] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:00.047202 8652 tablet_copy_client.cc:523] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:00.051082 8652 tablet_bootstrap.cc:492] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:13:00.058104 8652 tablet_bootstrap.cc:492] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:00.058480 8652 tablet_bootstrap.cc:492] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:13:00.058732 8652 ts_tablet_manager.cc:1367] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 08:13:00.059482 8652 raft_consensus.cc:348] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:00.059712 8652 raft_consensus.cc:729] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:13:00.059931 8652 consensus_queue.cc:260] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:00.064846 8652 ts_tablet_manager.cc:1398] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:13:00.066524 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session b8f581390da449ccb9e299011296405e-f4a205bf277d4770836dbbf0ec28d884 received from {username='slave'} at 127.27.157.193:54589
I20240503 08:13:00.066692 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session b8f581390da449ccb9e299011296405e-f4a205bf277d4770836dbbf0ec28d884 on tablet f4a205bf277d4770836dbbf0ec28d884 with peer b8f581390da449ccb9e299011296405e
I20240503 08:13:00.090186 4963 consensus_queue.cc:237] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:00.094058 4703 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:00.094272 4565 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:00.094780 8440 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:13:00.095127 4901 consensus_peers.cc:485] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dab60fefb4704ef19490fb3c76bf47ab. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.095135 8247 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:00.098671 8440 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:00.104106 4703 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:00.109232 4565 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:00.110502 1668 catalog_manager.cc:5484] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 4 to 5, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:00.132735 8560 consensus_queue.cc:1035] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.135958 8560 consensus_queue.cc:1035] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.148087 8560 consensus_queue.cc:1035] T 1aa23eb4acce4d6f8733eb80689c57e6 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.153635 8607 consensus_queue.cc:1035] T 8ae7520f758c47a39cfa3aaf59ebc674 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.161550 1812 consensus_queue.cc:237] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:00.168887 8616 consensus_queue.cc:1035] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.186764 8686 ts_tablet_manager.cc:902] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:13:00.193570 8617 consensus_queue.cc:1035] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.201469 8440 consensus_queue.cc:1035] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.245877 8686 tablet_copy_client.cc:308] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:13:00.251528 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet df0c416025c74075b6429d169fcdd58c from peer 43e2e01cdd314f9c872c4336ef74250a ({username='slave'} at 127.27.157.195:36783)
I20240503 08:13:00.251808 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet df0c416025c74075b6429d169fcdd58c from peer 43e2e01cdd314f9c872c4336ef74250a at {username='slave'} at 127.27.157.195:36783: session id = 43e2e01cdd314f9c872c4336ef74250a-df0c416025c74075b6429d169fcdd58c
I20240503 08:13:00.253836 8593 consensus_queue.cc:1035] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:00.254622 4855 tablet_copy_source_session.cc:205] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:00.254807 4700 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:00.256402 8629 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:00.256790 2203 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:00.256959 8686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df0c416025c74075b6429d169fcdd58c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:13:00.261626 1747 consensus_peers.cc:485] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fede923140fb45a5a4f93c0846b62595. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.262032 8629 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:00.265100 8686 tablet_copy_client.cc:783] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:00.265427 8686 tablet_copy_client.cc:647] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:00.276432 8686 tablet_copy_client.cc:523] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:00.293650 8629 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:00.294463 8686 tablet_bootstrap.cc:492] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:13:00.302480 2203 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:00.313060 1669 catalog_manager.cc:5484] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:00.336427 4698 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:00.346860 8686 tablet_bootstrap.cc:492] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:00.347306 8686 tablet_bootstrap.cc:492] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap complete.
I20240503 08:13:00.355540 8686 ts_tablet_manager.cc:1367] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.003s
I20240503 08:13:00.356352 8686 raft_consensus.cc:348] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:00.356624 8686 raft_consensus.cc:729] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: LEARNER
I20240503 08:13:00.356864 8686 consensus_queue.cc:260] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:00.362623 8686 ts_tablet_manager.cc:1398] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 08:13:00.363451 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 43e2e01cdd314f9c872c4336ef74250a-df0c416025c74075b6429d169fcdd58c received from {username='slave'} at 127.27.157.195:36783
I20240503 08:13:00.363608 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 43e2e01cdd314f9c872c4336ef74250a-df0c416025c74075b6429d169fcdd58c on tablet df0c416025c74075b6429d169fcdd58c with peer 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:13:00.383745 8708 ts_tablet_manager.cc:902] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:13:00.385879 8708 tablet_copy_client.cc:308] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.198:46559
I20240503 08:13:00.386755 4723 tablet_copy_service.cc:140] P 31f41542452b4b1eb994ff4223d039fc: Received BeginTabletCopySession request for tablet 01821d71873d4ad78d7b7f685161a229 from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:40623)
I20240503 08:13:00.386943 4723 tablet_copy_service.cc:161] P 31f41542452b4b1eb994ff4223d039fc: Beginning new tablet copy session on tablet 01821d71873d4ad78d7b7f685161a229 from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:40623: session id = 184c543cb05d49cc98ba1c56617be7a5-01821d71873d4ad78d7b7f685161a229
I20240503 08:13:00.389828 4723 tablet_copy_source_session.cc:205] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:00.391170 8708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01821d71873d4ad78d7b7f685161a229. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.392612 2072 consensus_queue.cc:237] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:00.396104 1942 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:00.396867 8486 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:13:00.398152 2008 consensus_peers.cc:485] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d4d06fdfc4e49d5b0790e2ee3b4cc83. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.398591 8708 tablet_copy_client.cc:783] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:00.398902 8708 tablet_copy_client.cc:647] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:00.399884 4698 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 08:13:00.401042 8616 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:00.401345 8708 tablet_copy_client.cc:523] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:00.401930 8486 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:00.402803 1942 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:00.408614 8708 tablet_bootstrap.cc:492] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:13:00.411604 1669 catalog_manager.cc:5484] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 4 to 5, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:00.421823 4698 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:00.427027 1812 raft_consensus.cc:1203] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:00.436460 8708 tablet_bootstrap.cc:492] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:00.436923 8708 tablet_bootstrap.cc:492] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:13:00.437247 8708 ts_tablet_manager.cc:1367] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.001s
I20240503 08:13:00.439610 8718 ts_tablet_manager.cc:902] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:13:00.440268 8708 raft_consensus.cc:348] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:00.440598 8708 raft_consensus.cc:729] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:13:00.440796 8708 consensus_queue.cc:260] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:00.443711 8708 ts_tablet_manager.cc:1398] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 08:13:00.444482 4723 tablet_copy_service.cc:342] P 31f41542452b4b1eb994ff4223d039fc: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-01821d71873d4ad78d7b7f685161a229 received from {username='slave'} at 127.27.157.196:40623
I20240503 08:13:00.444658 4723 tablet_copy_service.cc:433] P 31f41542452b4b1eb994ff4223d039fc: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-01821d71873d4ad78d7b7f685161a229 on tablet 01821d71873d4ad78d7b7f685161a229 with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:13:00.455806 8718 tablet_copy_client.cc:308] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:13:00.457355 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a ({username='slave'} at 127.27.157.194:46923)
I20240503 08:13:00.457590 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a at {username='slave'} at 127.27.157.194:46923: session id = 315f3f0bcbf447a4b93b1aafcfbf0f5a-4d3aaa1025ca4b0e832535d1ca3d5c4f
I20240503 08:13:00.460536 2093 tablet_copy_source_session.cc:205] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:00.464468 8718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.464639 8686 ts_tablet_manager.cc:902] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:13:00.471020 8718 tablet_copy_client.cc:783] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:00.471308 8718 tablet_copy_client.cc:647] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:00.471778 8686 tablet_copy_client.cc:308] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:13:00.472455 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet c6994370a532472893cc934337f3cb6e from peer 43e2e01cdd314f9c872c4336ef74250a ({username='slave'} at 127.27.157.195:53583)
I20240503 08:13:00.472646 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet c6994370a532472893cc934337f3cb6e from peer 43e2e01cdd314f9c872c4336ef74250a at {username='slave'} at 127.27.157.195:53583: session id = 43e2e01cdd314f9c872c4336ef74250a-c6994370a532472893cc934337f3cb6e
I20240503 08:13:00.473935 8718 tablet_copy_client.cc:523] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:00.474932 4988 tablet_copy_source_session.cc:205] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:00.476433 8686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6994370a532472893cc934337f3cb6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.478164 8718 tablet_bootstrap.cc:492] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:13:00.483033 2072 consensus_queue.cc:237] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:13:00.486312 8718 tablet_bootstrap.cc:492] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:00.486898 4698 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:00.487147 8718 tablet_bootstrap.cc:492] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap complete.
I20240503 08:13:00.487509 8718 ts_tablet_manager.cc:1367] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.003s
I20240503 08:13:00.487758 1943 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:00.487658 8486 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:00.488443 8718 raft_consensus.cc:348] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:13:00.488704 8718 raft_consensus.cc:729] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: LEARNER
I20240503 08:13:00.488937 8718 consensus_queue.cc:260] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
W20240503 08:13:00.489014 2010 consensus_peers.cc:485] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a92cb30e806b4e58a2d9f9cfa516bdf9. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.489607 8718 ts_tablet_manager.cc:1398] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:00.490550 8714 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:00.493606 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-4d3aaa1025ca4b0e832535d1ca3d5c4f received from {username='slave'} at 127.27.157.194:46923
I20240503 08:13:00.493757 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-4d3aaa1025ca4b0e832535d1ca3d5c4f on tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f with peer 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:13:00.498603 8686 tablet_copy_client.cc:783] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:00.498953 8686 tablet_copy_client.cc:647] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:00.502144 8686 tablet_copy_client.cc:523] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:00.507654 8686 tablet_bootstrap.cc:492] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:13:00.510046 8486 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:13:00.515084 4700 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:13:00.518080 8708 ts_tablet_manager.cc:902] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:13:00.521490 8686 tablet_bootstrap.cc:492] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:00.521963 8686 tablet_bootstrap.cc:492] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap complete.
I20240503 08:13:00.522266 8686 ts_tablet_manager.cc:1367] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20240503 08:13:00.523005 8686 raft_consensus.cc:348] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:00.523444 1669 catalog_manager.cc:5484] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:00.523298 8686 raft_consensus.cc:729] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: LEARNER
I20240503 08:13:00.526429 8686 consensus_queue.cc:260] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:00.527359 8686 ts_tablet_manager.cc:1398] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:13:00.528259 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session 43e2e01cdd314f9c872c4336ef74250a-c6994370a532472893cc934337f3cb6e received from {username='slave'} at 127.27.157.195:53583
I20240503 08:13:00.528402 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session 43e2e01cdd314f9c872c4336ef74250a-c6994370a532472893cc934337f3cb6e on tablet c6994370a532472893cc934337f3cb6e with peer 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:13:00.533286 8708 tablet_copy_client.cc:308] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:13:00.535472 8652 ts_tablet_manager.cc:902] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:13:00.536221 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet dab60fefb4704ef19490fb3c76bf47ab from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:40265)
I20240503 08:13:00.536363 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet dab60fefb4704ef19490fb3c76bf47ab from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:40265: session id = 184c543cb05d49cc98ba1c56617be7a5-dab60fefb4704ef19490fb3c76bf47ab
I20240503 08:13:00.536342 1943 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:13:00.545507 4988 tablet_copy_source_session.cc:205] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:00.546172 8652 tablet_copy_client.cc:308] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.196:42183
I20240503 08:13:00.547017 2224 tablet_copy_service.cc:140] P 184c543cb05d49cc98ba1c56617be7a5: Received BeginTabletCopySession request for tablet fe20c189635648f5b78905e8b9643bcd from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:56947)
I20240503 08:13:00.547178 2224 tablet_copy_service.cc:161] P 184c543cb05d49cc98ba1c56617be7a5: Beginning new tablet copy session on tablet fe20c189635648f5b78905e8b9643bcd from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:56947: session id = b8f581390da449ccb9e299011296405e-fe20c189635648f5b78905e8b9643bcd
I20240503 08:13:00.548189 8708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dab60fefb4704ef19490fb3c76bf47ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.551020 2224 tablet_copy_source_session.cc:205] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:00.552891 8708 tablet_copy_client.cc:783] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:00.553194 8708 tablet_copy_client.cc:647] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:00.553575 8652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe20c189635648f5b78905e8b9643bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.555680 8708 tablet_copy_client.cc:523] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:00.558261 8708 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:13:00.558631 8652 tablet_copy_client.cc:783] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:00.558895 8652 tablet_copy_client.cc:647] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:00.560889 8652 tablet_copy_client.cc:523] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:00.570885 8652 tablet_bootstrap.cc:492] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:13:00.577248 8652 tablet_bootstrap.cc:492] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:00.577625 8652 tablet_bootstrap.cc:492] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:13:00.577889 8652 ts_tablet_manager.cc:1367] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 08:13:00.578495 8652 raft_consensus.cc:348] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:00.578677 8652 raft_consensus.cc:729] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:13:00.578858 8652 consensus_queue.cc:260] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:00.579479 8652 ts_tablet_manager.cc:1398] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:00.580739 2224 tablet_copy_service.cc:342] P 184c543cb05d49cc98ba1c56617be7a5: Request end of tablet copy session b8f581390da449ccb9e299011296405e-fe20c189635648f5b78905e8b9643bcd received from {username='slave'} at 127.27.157.193:56947
I20240503 08:13:00.580853 2224 tablet_copy_service.cc:433] P 184c543cb05d49cc98ba1c56617be7a5: ending tablet copy session b8f581390da449ccb9e299011296405e-fe20c189635648f5b78905e8b9643bcd on tablet fe20c189635648f5b78905e8b9643bcd with peer b8f581390da449ccb9e299011296405e
I20240503 08:13:00.587894 8708 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:00.588287 8708 tablet_bootstrap.cc:492] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:13:00.588539 8708 ts_tablet_manager.cc:1367] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.004s
I20240503 08:13:00.589268 8708 raft_consensus.cc:348] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:00.589517 8708 raft_consensus.cc:729] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:13:00.589730 8708 consensus_queue.cc:260] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:00.590492 8708 ts_tablet_manager.cc:1398] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:00.592401 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-dab60fefb4704ef19490fb3c76bf47ab received from {username='slave'} at 127.27.157.196:40265
I20240503 08:13:00.592576 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-dab60fefb4704ef19490fb3c76bf47ab on tablet dab60fefb4704ef19490fb3c76bf47ab with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:13:00.593307 4700 consensus_queue.cc:237] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:00.597332 4565 raft_consensus.cc:1261] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:00.597890 1807 raft_consensus.cc:1261] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:00.597975 8705 consensus_queue.cc:1035] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:00.598402 4636 consensus_peers.cc:485] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb4e33ffc60043d9a50f8381e85b927a. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.598958 8624 consensus_queue.cc:1035] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:00.611752 8624 raft_consensus.cc:2932] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:00.612604 1807 raft_consensus.cc:2932] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:00.613520 4569 raft_consensus.cc:2932] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:00.623175 1669 catalog_manager.cc:5484] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:00.658165 2071 consensus_queue.cc:237] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:00.665136 2203 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:00.665163 4831 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:00.666115 8616 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:00.666529 8486 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:13:00.669389 2008 consensus_peers.cc:485] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 044d8e5a69444865980ffc06a6be9080. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.687510 8714 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:00.688490 2203 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:00.697325 1667 catalog_manager.cc:5484] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:00.698319 4831 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:00.737558 2203 consensus_queue.cc:237] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:13:00.740397 1807 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:00.741042 8027 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:00.741248 4831 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 08:13:00.741739 2140 consensus_peers.cc:485] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 913a3aacff964ac9a321fac187df33ee. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.742122 8027 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:00.745606 7924 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:13:00.750213 4831 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:13:00.757831 1669 catalog_manager.cc:5484] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 6 to 7, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:00.767807 8776 ts_tablet_manager.cc:902] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:13:00.769641 1807 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:13:00.791505 8776 tablet_copy_client.cc:308] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:41181
I20240503 08:13:00.792326 1833 tablet_copy_service.cc:140] P b8f581390da449ccb9e299011296405e: Received BeginTabletCopySession request for tablet fede923140fb45a5a4f93c0846b62595 from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:57509)
I20240503 08:13:00.792500 1833 tablet_copy_service.cc:161] P b8f581390da449ccb9e299011296405e: Beginning new tablet copy session on tablet fede923140fb45a5a4f93c0846b62595 from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:57509: session id = 00b61625d8a7460892c118a2570afda5-fede923140fb45a5a4f93c0846b62595
I20240503 08:13:00.795078 1833 tablet_copy_source_session.cc:205] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:00.795662 2071 raft_consensus.cc:1203] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:00.798566 8776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fede923140fb45a5a4f93c0846b62595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.807596 8776 tablet_copy_client.cc:783] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:00.808032 8776 tablet_copy_client.cc:647] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:00.810629 8776 tablet_copy_client.cc:523] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:00.815986 8776 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:13:00.816390 4963 consensus_queue.cc:237] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:13:00.819656 4565 raft_consensus.cc:1261] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:00.821791 1807 raft_consensus.cc:1261] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:00.822714 8247 consensus_queue.cc:1035] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:00.823187 4902 consensus_peers.cc:485] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 437037e814464cf5959153bcd3efc145. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:00.823390 8593 consensus_queue.cc:1035] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:00.827473 1943 raft_consensus.cc:1203] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:00.832080 8593 raft_consensus.cc:2932] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:13:00.838349 4569 raft_consensus.cc:2932] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:13:00.845194 8161 raft_consensus.cc:1050] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:13:00.846040 8161 consensus_queue.cc:237] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:00.849412 1669 catalog_manager.cc:5484] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:13:00.866334 1807 raft_consensus.cc:2932] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:13:00.884935 8708 ts_tablet_manager.cc:902] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:13:00.900796 4700 raft_consensus.cc:1261] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:00.903290 4967 raft_consensus.cc:1261] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:00.905561 8610 consensus_queue.cc:1035] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:00.906340 8451 consensus_queue.cc:1035] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:00.924927 1807 raft_consensus.cc:1261] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:00.928000 8451 consensus_queue.cc:1035] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:00.929391 2203 raft_consensus.cc:1203] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:00.940423 8161 raft_consensus.cc:2932] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:00.941666 8708 tablet_copy_client.cc:308] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:13:00.942476 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:51463)
I20240503 08:13:00.942662 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:51463: session id = 184c543cb05d49cc98ba1c56617be7a5-2d4d06fdfc4e49d5b0790e2ee3b4cc83
I20240503 08:13:00.945263 1809 raft_consensus.cc:1203] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:00.945776 2093 tablet_copy_source_session.cc:205] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:00.954123 4698 raft_consensus.cc:2932] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:00.954103 1809 raft_consensus.cc:2932] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:00.954504 4967 raft_consensus.cc:2932] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:00.963758 8708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:00.972939 2203 raft_consensus.cc:1203] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:13:00.975611 2071 raft_consensus.cc:1203] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:00.980844 8686 ts_tablet_manager.cc:902] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a: Initiating tablet copy from peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:13:00.983657 1667 catalog_manager.cc:5484] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:00.984325 8708 tablet_copy_client.cc:783] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:00.984608 8708 tablet_copy_client.cc:647] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:01.010823 4831 consensus_queue.cc:237] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:01.013542 4698 raft_consensus.cc:1261] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:01.013970 1807 raft_consensus.cc:1261] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:01.014375 8161 consensus_queue.cc:1035] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:01.015695 8161 consensus_queue.cc:1035] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:01.017906 8161 raft_consensus.cc:2932] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.019261 8708 tablet_copy_client.cc:523] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:01.020903 8776 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:01.020751 4698 raft_consensus.cc:2932] T f4a205bf277d4770836dbbf0ec28d884 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.021425 8776 tablet_bootstrap.cc:492] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:13:01.021761 8776 ts_tablet_manager.cc:1367] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.206s user 0.063s sys 0.038s
I20240503 08:13:01.022624 8776 raft_consensus.cc:348] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:01.022964 8776 raft_consensus.cc:729] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:13:01.023221 8776 consensus_queue.cc:260] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:01.025296 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4a205bf277d4770836dbbf0ec28d884 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:01.032750 8776 ts_tablet_manager.cc:1398] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20240503 08:13:01.033582 1833 tablet_copy_service.cc:342] P b8f581390da449ccb9e299011296405e: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-fede923140fb45a5a4f93c0846b62595 received from {username='slave'} at 127.27.157.200:57509
I20240503 08:13:01.033813 1833 tablet_copy_service.cc:433] P b8f581390da449ccb9e299011296405e: ending tablet copy session 00b61625d8a7460892c118a2570afda5-fede923140fb45a5a4f93c0846b62595 on tablet fede923140fb45a5a4f93c0846b62595 with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:13:01.041811 1809 raft_consensus.cc:2932] T f4a205bf277d4770836dbbf0ec28d884 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.041643 1668 catalog_manager.cc:5484] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:01.062355 8808 ts_tablet_manager.cc:902] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:13:01.064074 4945 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a205bf277d4770836dbbf0ec28d884 with delete_type TABLET_DATA_TOMBSTONED (TS 00b61625d8a7460892c118a2570afda5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47168
I20240503 08:13:01.068715 8810 tablet_replica.cc:330] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:13:01.072523 8810 raft_consensus.cc:2229] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:01.077962 8810 raft_consensus.cc:2258] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:01.081357 8810 ts_tablet_manager.cc:1875] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:01.084196 8708 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:13:01.101033 8686 tablet_copy_client.cc:308] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.198:46559
I20240503 08:13:01.102221 4723 tablet_copy_service.cc:140] P 31f41542452b4b1eb994ff4223d039fc: Received BeginTabletCopySession request for tablet eb4e33ffc60043d9a50f8381e85b927a from peer 43e2e01cdd314f9c872c4336ef74250a ({username='slave'} at 127.27.157.195:37471)
I20240503 08:13:01.102443 4723 tablet_copy_service.cc:161] P 31f41542452b4b1eb994ff4223d039fc: Beginning new tablet copy session on tablet eb4e33ffc60043d9a50f8381e85b927a from peer 43e2e01cdd314f9c872c4336ef74250a at {username='slave'} at 127.27.157.195:37471: session id = 43e2e01cdd314f9c872c4336ef74250a-eb4e33ffc60043d9a50f8381e85b927a
I20240503 08:13:01.109565 4723 tablet_copy_source_session.cc:205] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:01.109882 8708 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:01.110313 8808 tablet_copy_client.cc:308] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:13:01.111191 8686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4e33ffc60043d9a50f8381e85b927a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.111155 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet a92cb30e806b4e58a2d9f9cfa516bdf9 from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:52311)
I20240503 08:13:01.111805 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet a92cb30e806b4e58a2d9f9cfa516bdf9 from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:52311: session id = f24648c92c384f1ba349eee801f985e2-a92cb30e806b4e58a2d9f9cfa516bdf9
I20240503 08:13:01.112478 8810 ts_tablet_manager.cc:1888] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:01.112655 8810 log.cc:1196] T f4a205bf277d4770836dbbf0ec28d884 P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/f4a205bf277d4770836dbbf0ec28d884
I20240503 08:13:01.114991 2093 tablet_copy_source_session.cc:205] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:01.118957 8708 tablet_bootstrap.cc:492] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:13:01.119701 8808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a92cb30e806b4e58a2d9f9cfa516bdf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.120615 8708 ts_tablet_manager.cc:1367] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.003s
I20240503 08:13:01.121564 8708 raft_consensus.cc:348] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:01.121917 8708 raft_consensus.cc:729] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:13:01.122180 8708 consensus_queue.cc:260] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:01.129814 8708 ts_tablet_manager.cc:1398] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 08:13:01.131788 8686 tablet_copy_client.cc:783] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:01.132045 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet f4a205bf277d4770836dbbf0ec28d884 (table rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]) successfully deleted
I20240503 08:13:01.132964 8686 tablet_copy_client.cc:647] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:01.134742 8808 tablet_copy_client.cc:783] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:01.135569 8686 tablet_copy_client.cc:523] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:01.139729 8686 tablet_bootstrap.cc:492] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:13:01.140563 8808 tablet_copy_client.cc:647] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:01.143587 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-2d4d06fdfc4e49d5b0790e2ee3b4cc83 received from {username='slave'} at 127.27.157.196:51463
I20240503 08:13:01.143733 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-2d4d06fdfc4e49d5b0790e2ee3b4cc83 on tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:13:01.144518 8808 tablet_copy_client.cc:523] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:01.148602 8808 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:13:01.187692 8686 tablet_bootstrap.cc:492] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:01.188130 8686 tablet_bootstrap.cc:492] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap complete.
I20240503 08:13:01.188385 8686 ts_tablet_manager.cc:1367] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.049s user 0.023s sys 0.012s
I20240503 08:13:01.189110 8686 raft_consensus.cc:348] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:01.189363 8686 raft_consensus.cc:729] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: LEARNER
I20240503 08:13:01.189540 8686 consensus_queue.cc:260] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:01.193728 8686 ts_tablet_manager.cc:1398] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:13:01.199060 4723 tablet_copy_service.cc:342] P 31f41542452b4b1eb994ff4223d039fc: Request end of tablet copy session 43e2e01cdd314f9c872c4336ef74250a-eb4e33ffc60043d9a50f8381e85b927a received from {username='slave'} at 127.27.157.195:37471
I20240503 08:13:01.199236 4723 tablet_copy_service.cc:433] P 31f41542452b4b1eb994ff4223d039fc: ending tablet copy session 43e2e01cdd314f9c872c4336ef74250a-eb4e33ffc60043d9a50f8381e85b927a on tablet eb4e33ffc60043d9a50f8381e85b927a with peer 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:13:01.205013 8776 ts_tablet_manager.cc:902] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:13:01.229429 8808 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:01.229986 8808 tablet_bootstrap.cc:492] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:13:01.230326 8808 ts_tablet_manager.cc:1367] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.082s user 0.053s sys 0.001s
I20240503 08:13:01.231253 8808 raft_consensus.cc:348] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:13:01.231565 8808 raft_consensus.cc:729] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:13:01.231789 8808 consensus_queue.cc:260] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:13:01.232491 8808 ts_tablet_manager.cc:1398] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 08:13:01.233630 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-a92cb30e806b4e58a2d9f9cfa516bdf9 received from {username='slave'} at 127.27.157.199:52311
I20240503 08:13:01.233767 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session f24648c92c384f1ba349eee801f985e2-a92cb30e806b4e58a2d9f9cfa516bdf9 on tablet a92cb30e806b4e58a2d9f9cfa516bdf9 with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:13:01.262995 2203 raft_consensus.cc:1203] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 08:13:01.269106 8776 tablet_copy_client.cc:308] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:13:01.269862 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 044d8e5a69444865980ffc06a6be9080 from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:35695)
I20240503 08:13:01.270047 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 044d8e5a69444865980ffc06a6be9080 from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:35695: session id = 00b61625d8a7460892c118a2570afda5-044d8e5a69444865980ffc06a6be9080
I20240503 08:13:01.273144 2093 tablet_copy_source_session.cc:205] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:01.274856 8776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 044d8e5a69444865980ffc06a6be9080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.287505 8776 tablet_copy_client.cc:783] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:01.288115 8776 tablet_copy_client.cc:647] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:01.290704 8776 tablet_copy_client.cc:523] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:01.297138 8776 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:13:01.337452 8841 ts_tablet_manager.cc:902] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:13:01.343072 8314 raft_consensus.cc:1050] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:13:01.344484 8314 consensus_queue.cc:237] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:01.369243 8841 tablet_copy_client.cc:308] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.196:42183
I20240503 08:13:01.370240 2224 tablet_copy_service.cc:140] P 184c543cb05d49cc98ba1c56617be7a5: Received BeginTabletCopySession request for tablet 913a3aacff964ac9a321fac187df33ee from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:51901)
I20240503 08:13:01.370441 2224 tablet_copy_service.cc:161] P 184c543cb05d49cc98ba1c56617be7a5: Beginning new tablet copy session on tablet 913a3aacff964ac9a321fac187df33ee from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:51901: session id = 31f41542452b4b1eb994ff4223d039fc-913a3aacff964ac9a321fac187df33ee
I20240503 08:13:01.374543 2224 tablet_copy_source_session.cc:205] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:01.374876 8776 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:01.375422 8776 tablet_bootstrap.cc:492] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:13:01.375767 8776 ts_tablet_manager.cc:1367] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.079s user 0.052s sys 0.011s
I20240503 08:13:01.377110 4963 raft_consensus.cc:1203] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:01.377903 8841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913a3aacff964ac9a321fac187df33ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:01.377904 8776 raft_consensus.cc:348] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:01.378177 8776 raft_consensus.cc:729] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:13:01.378440 8776 consensus_queue.cc:260] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:01.380414 2204 raft_consensus.cc:1261] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.383056 8776 ts_tablet_manager.cc:1398] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:13:01.384264 8852 ts_tablet_manager.cc:902] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:13:01.384912 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-044d8e5a69444865980ffc06a6be9080 received from {username='slave'} at 127.27.157.200:35695
I20240503 08:13:01.385094 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 00b61625d8a7460892c118a2570afda5-044d8e5a69444865980ffc06a6be9080 on tablet 044d8e5a69444865980ffc06a6be9080 with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:13:01.388849 4569 raft_consensus.cc:1261] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.389544 8841 tablet_copy_client.cc:783] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:01.389969 8841 tablet_copy_client.cc:647] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:01.392671 7924 raft_consensus.cc:1050] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:13:01.393182 8705 consensus_queue.cc:1035] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.393841 7924 consensus_queue.cc:237] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:01.414433 8593 raft_consensus.cc:1050] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:13:01.415760 8593 consensus_queue.cc:237] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:01.431296 8705 consensus_queue.cc:1035] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.431952 2074 raft_consensus.cc:1203] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:01.435865 1943 raft_consensus.cc:1261] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.437285 8027 consensus_queue.cc:1035] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.437988 4700 raft_consensus.cc:1261] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.438616 8794 raft_consensus.cc:1050] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER 43e2e01cdd314f9c872c4336ef74250a to VOTER
I20240503 08:13:01.438762 8856 consensus_queue.cc:1035] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.439137 8617 raft_consensus.cc:1050] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a to VOTER
I20240503 08:13:01.439461 8794 consensus_queue.cc:237] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:01.439988 8617 consensus_queue.cc:237] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:13:01.450235 8610 raft_consensus.cc:1050] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 43e2e01cdd314f9c872c4336ef74250a to VOTER
I20240503 08:13:01.450603 1809 raft_consensus.cc:1261] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.451473 8610 consensus_queue.cc:237] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:01.455081 8027 consensus_queue.cc:1035] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.469236 4568 raft_consensus.cc:1261] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.468881 8027 raft_consensus.cc:2932] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.470036 8451 consensus_queue.cc:1035] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.479061 2071 raft_consensus.cc:1261] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.480044 8451 consensus_queue.cc:1035] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.488289 1809 raft_consensus.cc:2932] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.488287 1942 raft_consensus.cc:2932] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.494630 2073 raft_consensus.cc:1261] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.496800 4831 raft_consensus.cc:1261] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.498199 8786 consensus_queue.cc:1035] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.498099 1668 catalog_manager.cc:5484] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.499068 8786 consensus_queue.cc:1035] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.508858 4703 raft_consensus.cc:2932] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.519865 2203 consensus_queue.cc:237] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:01.523180 1943 raft_consensus.cc:1261] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:01.525759 8856 consensus_queue.cc:1035] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:01.533288 7924 raft_consensus.cc:2932] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.546914 8610 raft_consensus.cc:2932] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.547946 2074 raft_consensus.cc:2932] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.550585 8841 tablet_copy_client.cc:523] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:01.555243 8841 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:13:01.559243 4568 raft_consensus.cc:2932] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.562631 2073 raft_consensus.cc:1261] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.563666 1807 raft_consensus.cc:1261] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:01.566783 1942 raft_consensus.cc:2932] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.567266 1667 catalog_manager.cc:5484] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:01.570814 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe20c189635648f5b78905e8b9643bcd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:01.575304 8705 raft_consensus.cc:2932] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:01.566176 8873 consensus_queue.cc:1035] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20240503 08:13:01.581835 1809 raft_consensus.cc:2932] T fe20c189635648f5b78905e8b9643bcd P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.583103 2203 raft_consensus.cc:2932] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:01.586225 4834 consensus_queue.cc:237] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:01.586653 1668 catalog_manager.cc:5484] T fe20c189635648f5b78905e8b9643bcd P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:01.591902 4568 raft_consensus.cc:1261] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:01.592077 4700 raft_consensus.cc:1261] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.592726 8610 consensus_queue.cc:1035] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:01.593397 4833 raft_consensus.cc:1203] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:01.593765 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet fe20c189635648f5b78905e8b9643bcd with delete_type TABLET_DATA_TOMBSTONED (TS 31f41542452b4b1eb994ff4223d039fc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57094
I20240503 08:13:01.593814 8314 consensus_queue.cc:1035] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20240503 08:13:01.595541 4565 raft_consensus.cc:2932] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:01.595629 8161 raft_consensus.cc:2932] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.600103 4569 raft_consensus.cc:2932] T df0c416025c74075b6429d169fcdd58c P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.603799 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df0c416025c74075b6429d169fcdd58c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:01.605785 2074 raft_consensus.cc:1261] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:01.606626 8610 consensus_queue.cc:1035] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:01.611948 2071 raft_consensus.cc:2932] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.612864 1669 catalog_manager.cc:5484] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:01.622174 1670 catalog_manager.cc:5484] T df0c416025c74075b6429d169fcdd58c P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.627607 4681 tablet_service.cc:1514] Processing DeleteTablet for tablet df0c416025c74075b6429d169fcdd58c with delete_type TABLET_DATA_TOMBSTONED (TS 31f41542452b4b1eb994ff4223d039fc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57094
I20240503 08:13:01.643898 2073 raft_consensus.cc:2932] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:01.647806 4831 raft_consensus.cc:2932] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.671593 1942 raft_consensus.cc:1261] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.672762 4831 raft_consensus.cc:1261] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.672788 8714 consensus_queue.cc:1035] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.673417 8881 consensus_queue.cc:1035] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.676692 1670 catalog_manager.cc:5484] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.682353 8883 tablet_replica.cc:330] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:13:01.682634 8883 raft_consensus.cc:2229] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:01.683096 8883 raft_consensus.cc:2258] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:01.695547 8852 tablet_copy_client.cc:308] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:13:01.666543 8786 raft_consensus.cc:2932] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.701077 8617 raft_consensus.cc:2932] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:01.705349 4834 raft_consensus.cc:2932] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:01.708833 4967 raft_consensus.cc:1203] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:01.719893 1667 catalog_manager.cc:5484] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:01.722638 1942 raft_consensus.cc:2932] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:01.739568 8883 ts_tablet_manager.cc:1875] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:01.719844 4700 consensus_queue.cc:237] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:01.748535 2202 raft_consensus.cc:1261] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:01.749575 8884 consensus_queue.cc:1035] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:01.753482 8625 raft_consensus.cc:2932] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:01.755914 2073 consensus_queue.cc:237] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:13:01.758373 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01821d71873d4ad78d7b7f685161a229 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:01.761345 1943 raft_consensus.cc:1261] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:01.761469 2202 raft_consensus.cc:2932] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:01.763221 8714 consensus_queue.cc:1035] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:01.766887 8714 raft_consensus.cc:2932] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:01.768951 1668 catalog_manager.cc:5484] T 01821d71873d4ad78d7b7f685161a229 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:01.773741 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet 01821d71873d4ad78d7b7f685161a229 with delete_type TABLET_DATA_TOMBSTONED (TS 5f95b05d111e4090bbc59dec394ed0ab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39444
I20240503 08:13:01.778796 1943 raft_consensus.cc:2932] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:01.785373 2074 raft_consensus.cc:1261] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:01.787110 8705 consensus_queue.cc:1035] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:01.790830 2074 raft_consensus.cc:2932] T 01821d71873d4ad78d7b7f685161a229 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:01.795872 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:01.802064 1670 catalog_manager.cc:5484] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:01.811672 4831 raft_consensus.cc:1261] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:01.812238 8894 tablet_replica.cc:330] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:13:01.812386 8617 consensus_queue.cc:1035] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:01.812525 8894 raft_consensus.cc:2229] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:01.812908 8894 raft_consensus.cc:2258] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:01.815724 4834 raft_consensus.cc:2932] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:01.823051 4945 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f with delete_type TABLET_DATA_TOMBSTONED (TS 00b61625d8a7460892c118a2570afda5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47168
I20240503 08:13:01.823985 8894 ts_tablet_manager.cc:1875] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:01.830760 4965 raft_consensus.cc:1261] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.839548 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet 437037e814464cf5959153bcd3efc145 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a ({username='slave'} at 127.27.157.194:41645)
I20240503 08:13:01.840051 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet 437037e814464cf5959153bcd3efc145 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a at {username='slave'} at 127.27.157.194:41645: session id = 315f3f0bcbf447a4b93b1aafcfbf0f5a-437037e814464cf5959153bcd3efc145
I20240503 08:13:01.844645 8883 ts_tablet_manager.cc:1888] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:01.844810 8883 log.cc:1196] T fe20c189635648f5b78905e8b9643bcd P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/fe20c189635648f5b78905e8b9643bcd
I20240503 08:13:01.856751 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet fe20c189635648f5b78905e8b9643bcd (table rebalancer_extra_table_30 [id=b6256afd3f1a487c896a8ef180a92b10]) successfully deleted
I20240503 08:13:01.857492 4988 tablet_copy_source_session.cc:205] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:01.857916 8883 tablet_replica.cc:330] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:13:01.859136 8852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437037e814464cf5959153bcd3efc145. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 08:13:01.854491 4900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.197:42465, user_credentials={real_user=slave}} blocked reactor thread for 345547us
I20240503 08:13:01.862977 4568 raft_consensus.cc:1261] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.864275 4569 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:01.868021 8851 raft_consensus.cc:1050] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc: attempting to promote NON_VOTER 43e2e01cdd314f9c872c4336ef74250a to VOTER
I20240503 08:13:01.868917 8851 consensus_queue.cc:237] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:01.869918 8617 raft_consensus.cc:1050] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:13:01.870640 8617 consensus_queue.cc:237] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:01.880385 2203 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:01.881486 4698 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:01.894853 8894 ts_tablet_manager.cc:1888] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:01.900768 8247 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240503 08:13:01.903477 1943 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:01.904023 2203 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:01.904172 8714 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:01.904785 8714 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:01.904326 8894 log.cc:1196] T 01821d71873d4ad78d7b7f685161a229 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/01821d71873d4ad78d7b7f685161a229
I20240503 08:13:01.908128 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 01821d71873d4ad78d7b7f685161a229 (table rebalancer_extra_table_20 [id=f292cdedaf1b46d3a5d119cefab5e3c0]) successfully deleted
I20240503 08:13:01.912659 2073 raft_consensus.cc:1261] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.913694 8884 consensus_queue.cc:1035] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.928016 1809 raft_consensus.cc:1261] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.928848 4568 raft_consensus.cc:1261] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:01.929869 8884 consensus_queue.cc:1035] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.930449 8884 consensus_queue.cc:1035] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:01.945147 4698 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 08:13:01.945753 8714 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:01.948207 8852 tablet_copy_client.cc:783] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:01.948783 8852 tablet_copy_client.cc:647] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:01.955184 8870 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20240503 08:13:01.963584 8705 raft_consensus.cc:2932] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.964967 8879 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:01.969220 4568 raft_consensus.cc:2932] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.969805 4700 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:01.969941 2074 raft_consensus.cc:2932] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:01.981307 1670 catalog_manager.cc:5484] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:01.998967 8897 tablet_replica.cc:330] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:13:01.999251 8897 raft_consensus.cc:2229] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:01.999222 1667 catalog_manager.cc:5484] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 5 to 6, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.001040 2073 raft_consensus.cc:2932] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:02.003708 8897 raft_consensus.cc:2258] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:02.004402 8897 ts_tablet_manager.cc:1875] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:02.009191 2202 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.010067 1809 raft_consensus.cc:2932] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:02.026649 4700 consensus_queue.cc:237] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:02.029390 1807 raft_consensus.cc:1261] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:02.030170 8705 consensus_queue.cc:1035] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:02.031416 8852 tablet_copy_client.cc:523] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:02.035490 8895 consensus_queue.cc:1035] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:02.036496 8892 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:02.039996 8705 raft_consensus.cc:2932] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:02.044773 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb4e33ffc60043d9a50f8381e85b927a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:02.046425 8883 raft_consensus.cc:2229] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:02.046697 8883 raft_consensus.cc:2258] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:02.046236 1809 raft_consensus.cc:2932] T eb4e33ffc60043d9a50f8381e85b927a P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:02.047385 8883 ts_tablet_manager.cc:1875] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:02.049192 8852 tablet_bootstrap.cc:492] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:13:02.050318 8892 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.056056 4698 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.056192 1667 catalog_manager.cc:5484] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:02.061930 8617 raft_consensus.cc:1050] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:13:02.062469 8897 ts_tablet_manager.cc:1888] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:02.062618 8897 log.cc:1196] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/4d3aaa1025ca4b0e832535d1ca3d5c4f
I20240503 08:13:02.062917 8617 consensus_queue.cc:237] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:13:02.065699 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f (table rebalancer_extra_table_18 [id=8a48fca96c0e4fdfa7cd9f966b583e55]) successfully deleted
I20240503 08:13:02.069098 1668 catalog_manager.cc:5484] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 5 to 6, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:02.074903 2204 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.081768 4964 consensus_queue.cc:237] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:02.084033 4698 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:02.084856 8892 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:02.087685 8896 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.090144 2204 raft_consensus.cc:1261] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:02.090991 8892 consensus_queue.cc:1035] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:02.093125 2204 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.094249 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dab60fefb4704ef19490fb3c76bf47ab with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:02.097057 1669 catalog_manager.cc:5484] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 6 to 7, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:02.102977 4698 raft_consensus.cc:2932] T dab60fefb4704ef19490fb3c76bf47ab P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.109537 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet dab60fefb4704ef19490fb3c76bf47ab with delete_type TABLET_DATA_TOMBSTONED (TS 5f95b05d111e4090bbc59dec394ed0ab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39444
I20240503 08:13:02.110152 8894 tablet_replica.cc:330] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:13:02.110443 8894 raft_consensus.cc:2229] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:02.110810 8894 raft_consensus.cc:2258] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:02.111665 8894 ts_tablet_manager.cc:1875] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:02.115054 8913 raft_consensus.cc:1050] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:13:02.116113 1943 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.120147 2074 raft_consensus.cc:1261] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:02.121013 8851 consensus_queue.cc:1035] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:02.121554 8894 ts_tablet_manager.cc:1888] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 08:13:02.121773 8894 log.cc:1196] T dab60fefb4704ef19490fb3c76bf47ab P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/dab60fefb4704ef19490fb3c76bf47ab
I20240503 08:13:02.122800 2074 raft_consensus.cc:2932] T eb4e33ffc60043d9a50f8381e85b927a P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:02.123958 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet dab60fefb4704ef19490fb3c76bf47ab (table rebalancer_extra_table_4 [id=bc178b718fe049ae96bd1ff6f9794fba]) successfully deleted
I20240503 08:13:02.129364 8852 tablet_bootstrap.cc:492] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:02.129910 8852 tablet_bootstrap.cc:492] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap complete.
I20240503 08:13:02.130193 8852 ts_tablet_manager.cc:1367] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.001s
I20240503 08:13:02.142887 8913 consensus_queue.cc:237] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:02.145368 4831 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:02.146011 8880 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:02.150780 4568 raft_consensus.cc:2932] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:02.152674 4698 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:02.153391 8880 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:02.170696 1942 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:02.171912 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4e33ffc60043d9a50f8381e85b927a with delete_type TABLET_DATA_TOMBSTONED (TS 5f95b05d111e4090bbc59dec394ed0ab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39444
I20240503 08:13:02.172324 8894 tablet_replica.cc:330] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:13:02.172561 8894 raft_consensus.cc:2229] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:02.172485 8852 raft_consensus.cc:348] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:13:02.172717 8852 raft_consensus.cc:729] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: LEARNER
I20240503 08:13:02.172843 8894 raft_consensus.cc:2258] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:02.172883 8852 consensus_queue.cc:260] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:13:02.173408 8852 ts_tablet_manager.cc:1398] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20240503 08:13:02.174432 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-437037e814464cf5959153bcd3efc145 received from {username='slave'} at 127.27.157.194:41645
I20240503 08:13:02.174559 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-437037e814464cf5959153bcd3efc145 on tablet 437037e814464cf5959153bcd3efc145 with peer 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:13:02.176291 8894 ts_tablet_manager.cc:1875] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:02.180765 8880 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:02.184630 4700 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:02.185442 8916 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:02.191804 8894 ts_tablet_manager.cc:1888] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:02.192392 8894 log.cc:1196] T eb4e33ffc60043d9a50f8381e85b927a P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/eb4e33ffc60043d9a50f8381e85b927a
I20240503 08:13:02.193800 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet eb4e33ffc60043d9a50f8381e85b927a (table rebalancer_extra_table_24 [id=d79878310527447b9246772f2dcc1501]) successfully deleted
I20240503 08:13:02.196449 4964 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:02.198246 8914 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:02.200731 2073 consensus_queue.cc:237] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:02.203588 1942 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:02.195716 2204 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:02.204136 8617 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:02.204957 2204 raft_consensus.cc:1261] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:02.205595 8714 consensus_queue.cc:1035] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:02.206261 8879 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:02.206610 8883 ts_tablet_manager.cc:1888] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 08:13:02.206760 8883 log.cc:1196] T df0c416025c74075b6429d169fcdd58c P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/df0c416025c74075b6429d169fcdd58c
I20240503 08:13:02.206382 8913 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:02.208300 8714 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.211031 1941 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:02.211961 2201 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.217180 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet df0c416025c74075b6429d169fcdd58c (table rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]) successfully deleted
I20240503 08:13:02.220677 4834 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:02.221500 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:02.227967 1941 raft_consensus.cc:2932] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.235812 1667 catalog_manager.cc:5484] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:02.238499 1669 catalog_manager.cc:5484] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 6 to 7, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:02.249742 8913 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:02.253027 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 with delete_type TABLET_DATA_TOMBSTONED (TS 31f41542452b4b1eb994ff4223d039fc not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57094
I20240503 08:13:02.253568 8883 tablet_replica.cc:330] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:13:02.253849 8883 raft_consensus.cc:2229] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:02.254137 8883 raft_consensus.cc:2258] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:02.259241 8883 ts_tablet_manager.cc:1875] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:02.259069 4700 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
W20240503 08:13:02.261545 8920 log.cc:924] Time spent T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20240503 08:13:02.273146 8883 ts_tablet_manager.cc:1888] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 08:13:02.273296 8883 log.cc:1196] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/2d4d06fdfc4e49d5b0790e2ee3b4cc83
I20240503 08:13:02.276316 8880 raft_consensus.cc:1050] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:13:02.277241 8880 consensus_queue.cc:237] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:02.277319 1670 catalog_manager.cc:5484] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:02.272835 4700 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:02.273232 2204 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:02.287093 2073 consensus_queue.cc:237] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:13:02.289750 4831 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:02.290412 8714 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:02.291144 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83 (table rebalancer_extra_table_0 [id=b3d2ac4623d84f8faf0d7dbe7b38d679]) successfully deleted
I20240503 08:13:02.295624 8879 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:02.296788 4834 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:02.301976 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a92cb30e806b4e58a2d9f9cfa516bdf9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:02.304606 1667 catalog_manager.cc:5484] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 8 to 9, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:02.313143 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet a92cb30e806b4e58a2d9f9cfa516bdf9 with delete_type TABLET_DATA_TOMBSTONED (TS 315f3f0bcbf447a4b93b1aafcfbf0f5a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50100
I20240503 08:13:02.314340 4967 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:02.316435 8714 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:02.326025 8934 tablet_replica.cc:330] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: stopping tablet replica
I20240503 08:13:02.327167 8934 raft_consensus.cc:2229] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:02.330935 2201 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:02.331507 8714 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:02.332095 4831 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:02.333295 8880 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:02.343890 4698 raft_consensus.cc:1261] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:02.349506 8880 consensus_queue.cc:1035] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:02.356222 8934 raft_consensus.cc:2258] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:02.357743 8934 ts_tablet_manager.cc:1875] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:02.359076 4698 raft_consensus.cc:2932] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:02.362104 4964 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:02.390235 8714 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:02.394366 8934 ts_tablet_manager.cc:1888] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:02.394618 8934 log.cc:1196] T a92cb30e806b4e58a2d9f9cfa516bdf9 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/wals/a92cb30e806b4e58a2d9f9cfa516bdf9
I20240503 08:13:02.395771 1658 catalog_manager.cc:4830] TS 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): tablet a92cb30e806b4e58a2d9f9cfa516bdf9 (table rebalance_test_table_13 [id=f69a9daaaa3f44c28eac3e5c9d3b76d5]) successfully deleted
I20240503 08:13:02.396389 2202 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:02.401557 1668 catalog_manager.cc:5484] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:02.407243 2073 consensus_queue.cc:237] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:02.407645 4964 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:02.410506 4831 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:02.421224 1943 raft_consensus.cc:1203] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:02.449404 4967 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:02.451736 8880 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:02.454799 8714 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:02.460091 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 044d8e5a69444865980ffc06a6be9080 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:02.461293 4967 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:02.463136 1667 catalog_manager.cc:5484] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 8 to 9, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:02.469322 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 044d8e5a69444865980ffc06a6be9080 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42570
I20240503 08:13:02.485885 4834 raft_consensus.cc:1261] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:02.486945 8714 consensus_queue.cc:1035] T 044d8e5a69444865980ffc06a6be9080 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:02.488715 4834 raft_consensus.cc:2932] T 044d8e5a69444865980ffc06a6be9080 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:02.492194 8841 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:02.492756 8841 tablet_bootstrap.cc:492] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:13:02.493086 8841 ts_tablet_manager.cc:1367] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.938s user 0.050s sys 0.013s
I20240503 08:13:02.499567 8841 raft_consensus.cc:348] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:13:02.499900 8841 raft_consensus.cc:729] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:13:02.500136 8841 consensus_queue.cc:260] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:13:02.502440 8841 ts_tablet_manager.cc:1398] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240503 08:13:02.504288 2224 tablet_copy_service.cc:342] P 184c543cb05d49cc98ba1c56617be7a5: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-913a3aacff964ac9a321fac187df33ee received from {username='slave'} at 127.27.157.198:51901
I20240503 08:13:02.504487 2224 tablet_copy_service.cc:433] P 184c543cb05d49cc98ba1c56617be7a5: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-913a3aacff964ac9a321fac187df33ee on tablet 913a3aacff964ac9a321fac187df33ee with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:13:02.512722 8949 tablet_replica.cc:330] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:13:02.513087 8949 raft_consensus.cc:2229] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:02.513705 8949 raft_consensus.cc:2258] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:02.516444 8949 ts_tablet_manager.cc:1875] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:02.522784 8949 ts_tablet_manager.cc:1888] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:02.523017 8949 log.cc:1196] T 044d8e5a69444865980ffc06a6be9080 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/044d8e5a69444865980ffc06a6be9080
I20240503 08:13:02.524314 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 044d8e5a69444865980ffc06a6be9080 (table rebalance_test_table_5 [id=81cf492c7a7940a7bf625ec8e2798f29]) successfully deleted
I20240503 08:13:02.596855 4967 consensus_queue.cc:237] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:02.599198 2074 raft_consensus.cc:1261] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:02.601119 8891 consensus_queue.cc:1035] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:02.604887 8891 raft_consensus.cc:2932] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:02.606475 4568 raft_consensus.cc:1261] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:02.606261 2074 raft_consensus.cc:2932] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:02.607900 8247 consensus_queue.cc:1035] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:02.617337 4568 raft_consensus.cc:2932] T c6994370a532472893cc934337f3cb6e P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:02.622045 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6994370a532472893cc934337f3cb6e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:02.623485 1667 catalog_manager.cc:5484] T c6994370a532472893cc934337f3cb6e P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:02.629207 4813 tablet_service.cc:1514] Processing DeleteTablet for tablet c6994370a532472893cc934337f3cb6e with delete_type TABLET_DATA_TOMBSTONED (TS f24648c92c384f1ba349eee801f985e2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33952
I20240503 08:13:02.649585 8985 tablet_replica.cc:330] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:13:02.650048 8985 raft_consensus.cc:2229] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:02.650368 8985 raft_consensus.cc:2258] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:02.653753 8985 ts_tablet_manager.cc:1875] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:02.665159 8985 ts_tablet_manager.cc:1888] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:02.665416 8985 log.cc:1196] T c6994370a532472893cc934337f3cb6e P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/c6994370a532472893cc934337f3cb6e
I20240503 08:13:02.666441 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet c6994370a532472893cc934337f3cb6e (table rebalancer_extra_table_36 [id=f286720bc4ec482fbae83ba618630095]) successfully deleted
I20240503 08:13:02.702908 1807 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fede923140fb45a5a4f93c0846b62595"
dest_uuid: "b8f581390da449ccb9e299011296405e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41080
I20240503 08:13:02.703269 1807 raft_consensus.cc:595] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Received request to transfer leadership
I20240503 08:13:02.732679 4964 consensus_queue.cc:237] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:13:02.737507 2202 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:13:02.737792 1807 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:13:02.738425 8895 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 08:13:02.738773 4903 consensus_peers.cc:485] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3123fdbb5c945bd84f97fb4c35c5523. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:02.738844 8891 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:02.743288 8895 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:13:02.746321 8914 raft_consensus.cc:979] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: : Instructing follower 31f41542452b4b1eb994ff4223d039fc to start an election
I20240503 08:13:02.746537 8914 raft_consensus.cc:1067] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Signalling peer 31f41542452b4b1eb994ff4223d039fc to start an election
I20240503 08:13:02.747712 4698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fede923140fb45a5a4f93c0846b62595"
dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
from {username='slave'} at 127.27.157.193:41709
I20240503 08:13:02.747974 4698 raft_consensus.cc:482] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:02.748095 4698 raft_consensus.cc:3037] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:02.747927 2202 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:13:02.750972 4698 raft_consensus.cc:504] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:02.755003 1670 catalog_manager.cc:5484] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 7 to 8, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:02.761204 4698 leader_election.cc:288] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 election: Requested vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:13:02.761886 4964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fede923140fb45a5a4f93c0846b62595" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:13:02.762115 4964 raft_consensus.cc:3037] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:02.763850 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fede923140fb45a5a4f93c0846b62595" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:13:02.764109 1809 raft_consensus.cc:3032] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:02.764212 1809 raft_consensus.cc:729] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:13:02.764535 1809 consensus_queue.cc:260] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:02.764931 1809 raft_consensus.cc:3037] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:02.765533 4964 raft_consensus.cc:2454] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 2.
I20240503 08:13:02.775269 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fede923140fb45a5a4f93c0846b62595" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:13:02.775528 2202 raft_consensus.cc:3037] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:02.760711 1807 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } } }
I20240503 08:13:02.778693 1809 raft_consensus.cc:2454] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 2.
I20240503 08:13:02.779516 4637 leader_election.cc:302] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters:
I20240503 08:13:02.779923 8705 raft_consensus.cc:2781] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:02.780112 8705 raft_consensus.cc:686] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 2 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:13:02.780426 8705 consensus_queue.cc:237] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:02.784617 2202 raft_consensus.cc:2454] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 2.
I20240503 08:13:02.786507 1670 catalog_manager.cc:5484] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 1 to 2, leader changed from b8f581390da449ccb9e299011296405e (127.27.157.193) to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 2 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:02.829028 4831 consensus_queue.cc:237] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
W20240503 08:13:02.831880 4768 consensus_peers.cc:485] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4422e4f265f344039734893b759c2050. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:02.832556 2204 raft_consensus.cc:1261] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:02.833245 8610 consensus_queue.cc:1035] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:02.836344 2074 raft_consensus.cc:1261] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:02.837438 8610 raft_consensus.cc:2932] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:02.839795 8891 raft_consensus.cc:1050] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a to VOTER
I20240503 08:13:02.840767 8891 consensus_queue.cc:237] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:13:02.855121 1943 raft_consensus.cc:1261] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:02.856854 8895 consensus_queue.cc:1035] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:02.859431 9021 consensus_queue.cc:1035] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:02.873165 2074 raft_consensus.cc:2932] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:02.884963 1670 catalog_manager.cc:5484] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:02.890182 1809 raft_consensus.cc:1261] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:02.890924 8895 consensus_queue.cc:1035] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:02.895422 2204 raft_consensus.cc:2932] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:02.905573 4568 raft_consensus.cc:1261] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:02.906777 8247 consensus_queue.cc:1035] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:02.922852 8891 raft_consensus.cc:2932] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:02.925012 4698 raft_consensus.cc:1203] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:02.927278 1809 raft_consensus.cc:2932] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:02.928928 4698 consensus_queue.cc:237] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:02.935076 1943 raft_consensus.cc:2932] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:02.935258 1667 catalog_manager.cc:5484] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
W20240503 08:13:02.940832 4636 consensus_peers.cc:485] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebd7877c8faf409a8d042bef38f218aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:02.942409 4964 raft_consensus.cc:1261] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:02.942770 4565 raft_consensus.cc:1261] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:02.942998 8705 consensus_queue.cc:1035] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:02.943373 8625 consensus_queue.cc:1035] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:02.946309 8625 raft_consensus.cc:2932] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:02.946861 4964 consensus_queue.cc:237] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:13:02.946995 4967 raft_consensus.cc:2932] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:02.950016 4568 raft_consensus.cc:2932] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:02.951481 1809 raft_consensus.cc:1261] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:02.952077 1943 raft_consensus.cc:1261] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:02.953392 8895 consensus_queue.cc:1035] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:02.953866 8895 consensus_queue.cc:1035] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:02.960186 1667 catalog_manager.cc:5484] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:02.962566 8891 raft_consensus.cc:2932] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:02.979552 4569 raft_consensus.cc:2932] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:02.986691 1809 raft_consensus.cc:2932] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:02.997094 1670 catalog_manager.cc:5484] T 437037e814464cf5959153bcd3efc145 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:03.003316 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet 437037e814464cf5959153bcd3efc145 with delete_type TABLET_DATA_TOMBSTONED (TS 5f95b05d111e4090bbc59dec394ed0ab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39444
I20240503 08:13:03.004079 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 437037e814464cf5959153bcd3efc145 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:03.006888 1943 raft_consensus.cc:2932] T 437037e814464cf5959153bcd3efc145 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:03.008006 9046 tablet_replica.cc:330] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:13:03.008239 9046 raft_consensus.cc:2229] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:03.008488 9046 raft_consensus.cc:2258] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:03.010339 9046 ts_tablet_manager.cc:1875] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:03.020599 9046 ts_tablet_manager.cc:1888] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:03.020849 9046 log.cc:1196] T 437037e814464cf5959153bcd3efc145 P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/437037e814464cf5959153bcd3efc145
I20240503 08:13:03.021816 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet 437037e814464cf5959153bcd3efc145 (table rebalancer_extra_table_20 [id=f292cdedaf1b46d3a5d119cefab5e3c0]) successfully deleted
I20240503 08:13:03.029728 2204 consensus_queue.cc:237] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
W20240503 08:13:03.045112 2139 consensus_peers.cc:485] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4177c11467b49d49452ac10bd983627. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.046577 4831 raft_consensus.cc:1261] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:03.046607 4569 raft_consensus.cc:1261] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:03.047883 8873 consensus_queue.cc:1035] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:03.048799 8873 consensus_queue.cc:1035] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:03.062985 8873 raft_consensus.cc:2932] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:03.063586 4569 raft_consensus.cc:2932] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:03.073545 1667 catalog_manager.cc:5484] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:03.080385 4834 raft_consensus.cc:2932] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:03.096110 4967 consensus_queue.cc:237] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
W20240503 08:13:03.098927 4900 consensus_peers.cc:485] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 733bed28775e4779b74e92d8a395234f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.099081 2204 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:13:03.099143 1809 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 08:13:03.099782 8874 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:03.100373 8874 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:03.104310 8874 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:13:03.105057 1809 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:13:03.105188 2204 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:13:03.116463 1667 catalog_manager.cc:5484] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 7 to 8, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:03.130167 9059 ts_tablet_manager.cc:902] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:13:03.137493 9059 tablet_copy_client.cc:308] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:13:03.138293 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet f3123fdbb5c945bd84f97fb4c35c5523 from peer 31f41542452b4b1eb994ff4223d039fc ({username='slave'} at 127.27.157.198:56537)
I20240503 08:13:03.138496 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet f3123fdbb5c945bd84f97fb4c35c5523 from peer 31f41542452b4b1eb994ff4223d039fc at {username='slave'} at 127.27.157.198:56537: session id = 31f41542452b4b1eb994ff4223d039fc-f3123fdbb5c945bd84f97fb4c35c5523
I20240503 08:13:03.141606 4988 tablet_copy_source_session.cc:205] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:03.143266 9059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3123fdbb5c945bd84f97fb4c35c5523. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.147691 1943 consensus_queue.cc:237] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:03.149777 9059 tablet_copy_client.cc:783] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:03.150117 9059 tablet_copy_client.cc:647] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:03.152845 2073 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:03.154083 8923 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:03.155557 4569 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:03.157145 9059 tablet_copy_client.cc:523] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:03.156950 9071 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:13:03.158978 1879 consensus_peers.cc:485] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 674e558bf5db4f169f8cdcb08c40f5fb. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.161684 9059 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap starting.
I20240503 08:13:03.171437 7964 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:03.172431 2073 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:03.179718 1667 catalog_manager.cc:5484] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 6 to 7, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:03.181366 4565 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:03.216987 1809 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:03.217844 8884 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:03.222496 2204 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:03.223135 8884 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:03.231223 9059 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:03.231765 9059 tablet_bootstrap.cc:492] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc: Bootstrap complete.
I20240503 08:13:03.232087 9059 ts_tablet_manager.cc:1367] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc: Time spent bootstrapping tablet: real 0.071s user 0.057s sys 0.004s
I20240503 08:13:03.233060 9059 raft_consensus.cc:348] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:13:03.233331 9059 raft_consensus.cc:729] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Initialized, Role: LEARNER
I20240503 08:13:03.233568 9059 consensus_queue.cc:260] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: NON_VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: true } }
I20240503 08:13:03.235747 9059 ts_tablet_manager.cc:1398] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:03.236546 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session 31f41542452b4b1eb994ff4223d039fc-f3123fdbb5c945bd84f97fb4c35c5523 received from {username='slave'} at 127.27.157.198:56537
I20240503 08:13:03.236694 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session 31f41542452b4b1eb994ff4223d039fc-f3123fdbb5c945bd84f97fb4c35c5523 on tablet f3123fdbb5c945bd84f97fb4c35c5523 with peer 31f41542452b4b1eb994ff4223d039fc
I20240503 08:13:03.259865 1943 consensus_queue.cc:237] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:13:03.265304 4967 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:03.270035 9082 ts_tablet_manager.cc:902] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:13:03.269912 8705 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:03.299764 2204 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:03.300252 4698 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 08:13:03.300506 9071 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 08:13:03.301335 1880 consensus_peers.cc:485] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c37418ac97343d1931a8e1239218a38. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:03.302701 8923 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:03.324090 8923 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:13:03.328367 2201 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:13:03.331859 4700 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:13:03.338069 1667 catalog_manager.cc:5484] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 6 to 7, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:03.385807 9082 tablet_copy_client.cc:308] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:13:03.386709 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet 4422e4f265f344039734893b759c2050 from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:52427)
I20240503 08:13:03.386965 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet 4422e4f265f344039734893b759c2050 from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:52427: session id = 00b61625d8a7460892c118a2570afda5-4422e4f265f344039734893b759c2050
I20240503 08:13:03.395421 4855 tablet_copy_source_session.cc:205] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:03.397328 9082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4422e4f265f344039734893b759c2050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.400230 9098 ts_tablet_manager.cc:902] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a: Initiating tablet copy from peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:13:03.406536 9082 tablet_copy_client.cc:783] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:03.406919 9082 tablet_copy_client.cc:647] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:03.411156 9082 tablet_copy_client.cc:523] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:03.414711 9082 tablet_bootstrap.cc:492] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:13:03.425558 9098 tablet_copy_client.cc:308] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.198:46559
I20240503 08:13:03.428025 4723 tablet_copy_service.cc:140] P 31f41542452b4b1eb994ff4223d039fc: Received BeginTabletCopySession request for tablet ebd7877c8faf409a8d042bef38f218aa from peer 43e2e01cdd314f9c872c4336ef74250a ({username='slave'} at 127.27.157.195:37471)
I20240503 08:13:03.428282 4723 tablet_copy_service.cc:161] P 31f41542452b4b1eb994ff4223d039fc: Beginning new tablet copy session on tablet ebd7877c8faf409a8d042bef38f218aa from peer 43e2e01cdd314f9c872c4336ef74250a at {username='slave'} at 127.27.157.195:37471: session id = 43e2e01cdd314f9c872c4336ef74250a-ebd7877c8faf409a8d042bef38f218aa
I20240503 08:13:03.431756 4723 tablet_copy_source_session.cc:205] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:03.432231 9082 tablet_bootstrap.cc:492] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:03.432775 9082 tablet_bootstrap.cc:492] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:13:03.433064 9082 ts_tablet_manager.cc:1367] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.011s
I20240503 08:13:03.434041 9082 raft_consensus.cc:348] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:03.434302 9082 raft_consensus.cc:729] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:13:03.435216 9098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd7877c8faf409a8d042bef38f218aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.435101 9082 consensus_queue.cc:260] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:03.435765 9082 ts_tablet_manager.cc:1398] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:03.437445 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-4422e4f265f344039734893b759c2050 received from {username='slave'} at 127.27.157.200:52427
I20240503 08:13:03.437629 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 00b61625d8a7460892c118a2570afda5-4422e4f265f344039734893b759c2050 on tablet 4422e4f265f344039734893b759c2050 with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:13:03.443909 9098 tablet_copy_client.cc:783] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:03.444696 9098 tablet_copy_client.cc:647] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:03.446568 9098 tablet_copy_client.cc:523] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:03.450743 9098 tablet_bootstrap.cc:492] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:13:03.456295 9098 tablet_bootstrap.cc:492] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:03.456737 9098 tablet_bootstrap.cc:492] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap complete.
I20240503 08:13:03.457007 9098 ts_tablet_manager.cc:1367] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 08:13:03.457777 9098 raft_consensus.cc:348] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:03.458067 9098 raft_consensus.cc:729] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: LEARNER
I20240503 08:13:03.458333 9098 consensus_queue.cc:260] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:03.462127 9098 ts_tablet_manager.cc:1398] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 08:13:03.462989 4723 tablet_copy_service.cc:342] P 31f41542452b4b1eb994ff4223d039fc: Request end of tablet copy session 43e2e01cdd314f9c872c4336ef74250a-ebd7877c8faf409a8d042bef38f218aa received from {username='slave'} at 127.27.157.195:37471
I20240503 08:13:03.463202 4723 tablet_copy_service.cc:433] P 31f41542452b4b1eb994ff4223d039fc: ending tablet copy session 43e2e01cdd314f9c872c4336ef74250a-ebd7877c8faf409a8d042bef38f218aa on tablet ebd7877c8faf409a8d042bef38f218aa with peer 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:13:03.542418 8873 raft_consensus.cc:1050] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:13:03.543454 8873 consensus_queue.cc:237] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:13:03.549207 4834 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:03.549261 4698 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc [term 1 LEARNER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:03.550107 8873 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:03.550472 1809 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:03.550956 8873 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:03.551682 8869 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:03.571368 8869 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:03.578830 1809 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:03.583127 4698 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:03.584601 1670 catalog_manager.cc:5484] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 7 to 8, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:03.585114 4834 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:03.591929 4700 raft_consensus.cc:1203] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:13:03.601883 2201 consensus_queue.cc:237] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:13:03.604547 4698 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:03.605156 9151 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:03.609092 8873 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:03.611536 4834 raft_consensus.cc:1261] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:03.612303 9048 consensus_queue.cc:1035] T 913a3aacff964ac9a321fac187df33ee P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:03.613811 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 913a3aacff964ac9a321fac187df33ee with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:03.614436 4698 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:03.621161 1667 catalog_manager.cc:5484] T 913a3aacff964ac9a321fac187df33ee P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 8 to 9, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:03.621373 4834 raft_consensus.cc:2932] T 913a3aacff964ac9a321fac187df33ee P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:03.625018 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet 913a3aacff964ac9a321fac187df33ee with delete_type TABLET_DATA_TOMBSTONED (TS b8f581390da449ccb9e299011296405e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48256
I20240503 08:13:03.625774 9152 tablet_replica.cc:330] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:13:03.626183 9152 raft_consensus.cc:2229] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:03.626886 9152 raft_consensus.cc:2258] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:03.633044 9152 ts_tablet_manager.cc:1875] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:03.641894 9152 ts_tablet_manager.cc:1888] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:03.642241 9152 log.cc:1196] T 913a3aacff964ac9a321fac187df33ee P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/913a3aacff964ac9a321fac187df33ee
I20240503 08:13:03.643522 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 913a3aacff964ac9a321fac187df33ee (table rebalance_test_table_19 [id=37a3ac4b4467489788ff2543c2b3a179]) successfully deleted
I20240503 08:13:03.655124 9156 ts_tablet_manager.cc:902] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:13:03.657279 4964 raft_consensus.cc:1203] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:03.702293 9162 ts_tablet_manager.cc:902] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:13:03.712383 9156 tablet_copy_client.cc:308] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.196:42183
I20240503 08:13:03.713160 2224 tablet_copy_service.cc:140] P 184c543cb05d49cc98ba1c56617be7a5: Received BeginTabletCopySession request for tablet c4177c11467b49d49452ac10bd983627 from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:56947)
I20240503 08:13:03.713347 2224 tablet_copy_service.cc:161] P 184c543cb05d49cc98ba1c56617be7a5: Beginning new tablet copy session on tablet c4177c11467b49d49452ac10bd983627 from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:56947: session id = b8f581390da449ccb9e299011296405e-c4177c11467b49d49452ac10bd983627
I20240503 08:13:03.716413 2224 tablet_copy_source_session.cc:205] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:03.720333 9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4177c11467b49d49452ac10bd983627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.720234 4698 consensus_queue.cc:237] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:03.722258 4964 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:03.724176 8625 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:03.725062 2201 raft_consensus.cc:1261] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:03.726089 8625 consensus_queue.cc:1035] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:03.728142 9156 tablet_copy_client.cc:783] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:03.728430 9156 tablet_copy_client.cc:647] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:03.729319 8625 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:03.733052 9173 ts_tablet_manager.cc:902] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:13:03.732792 4964 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:03.734967 9156 tablet_copy_client.cc:523] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:03.742084 9156 tablet_bootstrap.cc:492] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:13:03.745465 2201 raft_consensus.cc:2932] T fede923140fb45a5a4f93c0846b62595 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:03.750036 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fede923140fb45a5a4f93c0846b62595 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:03.760146 1663 catalog_manager.cc:5484] T fede923140fb45a5a4f93c0846b62595 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 8 to 10, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New cstate: current_term: 2 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:03.771561 9173 tablet_copy_client.cc:308] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:13:03.772696 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 674e558bf5db4f169f8cdcb08c40f5fb from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:57653)
I20240503 08:13:03.772858 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 674e558bf5db4f169f8cdcb08c40f5fb from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:57653: session id = b8f581390da449ccb9e299011296405e-674e558bf5db4f169f8cdcb08c40f5fb
I20240503 08:13:03.775810 1964 tablet_copy_source_session.cc:205] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:03.776968 9162 tablet_copy_client.cc:308] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:13:03.777554 9173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674e558bf5db4f169f8cdcb08c40f5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.781790 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet 733bed28775e4779b74e92d8a395234f from peer 5f95b05d111e4090bbc59dec394ed0ab ({username='slave'} at 127.27.157.197:45051)
I20240503 08:13:03.782007 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet 733bed28775e4779b74e92d8a395234f from peer 5f95b05d111e4090bbc59dec394ed0ab at {username='slave'} at 127.27.157.197:45051: session id = 5f95b05d111e4090bbc59dec394ed0ab-733bed28775e4779b74e92d8a395234f
I20240503 08:13:03.784951 4988 tablet_copy_source_session.cc:205] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:03.788002 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet fede923140fb45a5a4f93c0846b62595 with delete_type TABLET_DATA_TOMBSTONED (TS b8f581390da449ccb9e299011296405e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48256
I20240503 08:13:03.788368 9152 tablet_replica.cc:330] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:13:03.788594 9152 raft_consensus.cc:2229] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:03.788787 9152 raft_consensus.cc:2258] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:03.788784 9185 ts_tablet_manager.cc:902] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:13:03.788959 9162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733bed28775e4779b74e92d8a395234f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.790220 9152 ts_tablet_manager.cc:1875] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:03.794696 9173 tablet_copy_client.cc:783] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:03.794994 9173 tablet_copy_client.cc:647] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:03.796329 9156 tablet_bootstrap.cc:492] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:03.797358 9156 tablet_bootstrap.cc:492] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:13:03.799261 9156 ts_tablet_manager.cc:1367] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.010s
I20240503 08:13:03.800124 9156 raft_consensus.cc:348] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:03.800392 9156 raft_consensus.cc:729] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:13:03.800606 9156 consensus_queue.cc:260] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:03.801193 9156 ts_tablet_manager.cc:1398] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:03.801903 2224 tablet_copy_service.cc:342] P 184c543cb05d49cc98ba1c56617be7a5: Request end of tablet copy session b8f581390da449ccb9e299011296405e-c4177c11467b49d49452ac10bd983627 received from {username='slave'} at 127.27.157.193:56947
I20240503 08:13:03.802043 2224 tablet_copy_service.cc:433] P 184c543cb05d49cc98ba1c56617be7a5: ending tablet copy session b8f581390da449ccb9e299011296405e-c4177c11467b49d49452ac10bd983627 on tablet c4177c11467b49d49452ac10bd983627 with peer b8f581390da449ccb9e299011296405e
I20240503 08:13:03.811394 9173 tablet_copy_client.cc:523] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:03.812975 9162 tablet_copy_client.cc:783] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:03.813308 9162 tablet_copy_client.cc:647] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:03.815209 9162 tablet_copy_client.cc:523] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:03.820935 9152 ts_tablet_manager.cc:1888] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:13:03.821836 9162 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap starting.
I20240503 08:13:03.823315 9173 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:13:03.839478 9152 log.cc:1196] T fede923140fb45a5a4f93c0846b62595 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/fede923140fb45a5a4f93c0846b62595
I20240503 08:13:03.840736 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet fede923140fb45a5a4f93c0846b62595 (table rebalance_test_table_3 [id=19626a36f6c54bd7b459b0de14aadbfc]) successfully deleted
I20240503 08:13:03.861101 9185 tablet_copy_client.cc:308] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:13:03.861778 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 7c37418ac97343d1931a8e1239218a38 from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:51509)
I20240503 08:13:03.861944 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 7c37418ac97343d1931a8e1239218a38 from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:51509: session id = f24648c92c384f1ba349eee801f985e2-7c37418ac97343d1931a8e1239218a38
I20240503 08:13:03.869861 1964 tablet_copy_source_session.cc:205] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:03.871892 9185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c37418ac97343d1931a8e1239218a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:03.884996 9185 tablet_copy_client.cc:783] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:03.885365 9185 tablet_copy_client.cc:647] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:03.889935 9185 tablet_copy_client.cc:523] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:03.910604 9185 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:13:03.913681 9162 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:03.914268 9162 tablet_bootstrap.cc:492] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab: Bootstrap complete.
I20240503 08:13:03.914695 9162 ts_tablet_manager.cc:1367] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab: Time spent bootstrapping tablet: real 0.093s user 0.046s sys 0.013s
I20240503 08:13:03.916271 9162 raft_consensus.cc:348] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:13:03.916611 9162 raft_consensus.cc:729] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5f95b05d111e4090bbc59dec394ed0ab, State: Initialized, Role: LEARNER
I20240503 08:13:03.916875 9162 consensus_queue.cc:260] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:13:03.917546 9162 ts_tablet_manager.cc:1398] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab: Time spent starting tablet: real 0.003s user 0.001s sys 0.004s
I20240503 08:13:03.920384 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-733bed28775e4779b74e92d8a395234f received from {username='slave'} at 127.27.157.197:45051
I20240503 08:13:03.920542 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session 5f95b05d111e4090bbc59dec394ed0ab-733bed28775e4779b74e92d8a395234f on tablet 733bed28775e4779b74e92d8a395234f with peer 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:03.993242 9173 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:03.993803 9173 tablet_bootstrap.cc:492] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:13:03.994096 9173 ts_tablet_manager.cc:1367] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.171s user 0.054s sys 0.007s
I20240503 08:13:03.998052 9173 raft_consensus.cc:348] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:03.998337 9173 raft_consensus.cc:729] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:13:03.998528 9173 consensus_queue.cc:260] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:03.999276 9173 ts_tablet_manager.cc:1398] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 08:13:04.000384 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session b8f581390da449ccb9e299011296405e-674e558bf5db4f169f8cdcb08c40f5fb received from {username='slave'} at 127.27.157.193:57653
I20240503 08:13:04.000512 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session b8f581390da449ccb9e299011296405e-674e558bf5db4f169f8cdcb08c40f5fb on tablet 674e558bf5db4f169f8cdcb08c40f5fb with peer b8f581390da449ccb9e299011296405e
I20240503 08:13:04.015349 9185 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:04.016019 9185 tablet_bootstrap.cc:492] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:13:04.016834 9185 ts_tablet_manager.cc:1367] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.106s user 0.067s sys 0.030s
I20240503 08:13:04.017866 2073 raft_consensus.cc:1203] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:04.017951 9185 raft_consensus.cc:348] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:13:04.018352 9185 raft_consensus.cc:729] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:13:04.018604 9185 consensus_queue.cc:260] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:13:04.019219 9185 ts_tablet_manager.cc:1398] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:04.021519 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-7c37418ac97343d1931a8e1239218a38 received from {username='slave'} at 127.27.157.199:51509
I20240503 08:13:04.021674 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session f24648c92c384f1ba349eee801f985e2-7c37418ac97343d1931a8e1239218a38 on tablet 7c37418ac97343d1931a8e1239218a38 with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:13:04.036031 1807 raft_consensus.cc:1203] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:04.063635 4565 raft_consensus.cc:1203] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 08:13:04.076910 4698 consensus_queue.cc:237] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:04.079515 8874 raft_consensus.cc:1050] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER 31f41542452b4b1eb994ff4223d039fc to VOTER
I20240503 08:13:04.080324 8874 consensus_queue.cc:237] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:13:04.084101 1942 raft_consensus.cc:1261] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:04.084314 4637 consensus_peers.cc:485] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c02a6bb7f7d4b42a6af774f342b0398. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:04.084652 8705 consensus_queue.cc:1035] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:04.084844 2201 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:04.086220 8247 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:04.088979 8705 raft_consensus.cc:2932] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:04.093658 4834 raft_consensus.cc:1261] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:04.093315 1942 raft_consensus.cc:2932] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:04.095217 8625 consensus_queue.cc:1035] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:04.102802 1667 catalog_manager.cc:5484] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:04.110062 1807 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:04.111034 8895 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:04.121193 4698 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc [term 2 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:04.125669 8895 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:04.137305 8247 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:04.142398 4700 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:04.143039 2201 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:04.157305 1668 catalog_manager.cc:5484] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 8 to 9, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:04.162736 1809 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:04.173002 4834 raft_consensus.cc:2932] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:04.173393 4964 consensus_queue.cc:237] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } }
I20240503 08:13:04.175796 2201 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:04.175848 4698 raft_consensus.cc:1261] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:04.176828 8247 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:04.184789 8247 consensus_queue.cc:1035] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:04.188077 8247 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:04.191758 4699 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:04.193107 2201 raft_consensus.cc:2932] T f3123fdbb5c945bd84f97fb4c35c5523 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:04.202601 1667 catalog_manager.cc:5484] T f3123fdbb5c945bd84f97fb4c35c5523 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 9 to 10, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } }
I20240503 08:13:04.207216 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f3123fdbb5c945bd84f97fb4c35c5523 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:04.213315 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet f3123fdbb5c945bd84f97fb4c35c5523 with delete_type TABLET_DATA_TOMBSTONED (TS b8f581390da449ccb9e299011296405e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48256
I20240503 08:13:04.214102 9152 tablet_replica.cc:330] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:13:04.214329 9152 raft_consensus.cc:2229] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:04.214507 9152 raft_consensus.cc:2258] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:04.215624 9152 ts_tablet_manager.cc:1875] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:04.221664 9152 ts_tablet_manager.cc:1888] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:13:04.221913 9152 log.cc:1196] T f3123fdbb5c945bd84f97fb4c35c5523 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/f3123fdbb5c945bd84f97fb4c35c5523
I20240503 08:13:04.222761 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet f3123fdbb5c945bd84f97fb4c35c5523 (table rebalance_test_table_11 [id=4e422caa613e4bae95fb8412d4e942ca]) successfully deleted
I20240503 08:13:04.230773 9028 raft_consensus.cc:1050] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:13:04.231662 9028 consensus_queue.cc:237] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:04.243510 4964 raft_consensus.cc:1261] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:04.244268 2073 raft_consensus.cc:1261] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:04.244362 9028 consensus_queue.cc:1035] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:04.244844 2201 raft_consensus.cc:1261] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:04.255435 4834 raft_consensus.cc:1203] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:04.282615 8866 consensus_queue.cc:1035] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:04.289333 9025 consensus_queue.cc:1035] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:04.306308 1807 raft_consensus.cc:1203] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 08:13:04.322883 4964 raft_consensus.cc:2932] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:04.321822 9241 raft_consensus.cc:2932] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:04.333530 1668 catalog_manager.cc:5484] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:04.335601 2073 raft_consensus.cc:2932] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:04.346954 2204 raft_consensus.cc:2932] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:04.363278 4834 consensus_queue.cc:237] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:04.365492 4964 raft_consensus.cc:1261] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.365625 2073 raft_consensus.cc:1261] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.369941 9241 consensus_queue.cc:1035] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.370707 9241 consensus_queue.cc:1035] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.372968 9239 raft_consensus.cc:2932] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:04.376325 2073 raft_consensus.cc:2932] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:04.379477 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4422e4f265f344039734893b759c2050 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:04.388770 1668 catalog_manager.cc:5484] T 4422e4f265f344039734893b759c2050 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 3 to 4, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:04.398820 4964 raft_consensus.cc:2932] T 4422e4f265f344039734893b759c2050 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:04.410374 8873 raft_consensus.cc:1050] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:13:04.411113 8873 consensus_queue.cc:237] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:04.414002 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 4422e4f265f344039734893b759c2050 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42570
I20240503 08:13:04.428030 9253 tablet_replica.cc:330] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:13:04.428267 9253 raft_consensus.cc:2229] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:04.428481 9253 raft_consensus.cc:2258] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:04.430647 1809 raft_consensus.cc:1261] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:04.430720 4565 raft_consensus.cc:1261] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:04.430758 9253 ts_tablet_manager.cc:1875] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:04.431375 9151 consensus_queue.cc:1035] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:04.431480 4834 raft_consensus.cc:1261] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:04.431872 9151 consensus_queue.cc:1035] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:04.432276 9048 consensus_queue.cc:1035] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:04.437865 9253 ts_tablet_manager.cc:1888] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:04.438163 9253 log.cc:1196] T 4422e4f265f344039734893b759c2050 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/4422e4f265f344039734893b759c2050
I20240503 08:13:04.439977 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 4422e4f265f344039734893b759c2050 (table rebalancer_extra_table_28 [id=a4d2142b44bc4afa85d24f2a4c14b381]) successfully deleted
I20240503 08:13:04.448926 9254 raft_consensus.cc:2932] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.449885 1809 raft_consensus.cc:2932] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.454766 4834 raft_consensus.cc:2932] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.455727 4565 raft_consensus.cc:2932] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.461216 1668 catalog_manager.cc:5484] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:04.473938 2201 consensus_queue.cc:237] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:04.479071 9173 ts_tablet_manager.cc:902] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:13:04.481470 8884 raft_consensus.cc:1050] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc: attempting to promote NON_VOTER 43e2e01cdd314f9c872c4336ef74250a to VOTER
I20240503 08:13:04.482427 8884 consensus_queue.cc:237] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:04.484886 1809 raft_consensus.cc:1261] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.485607 9254 consensus_queue.cc:1035] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.490569 4565 raft_consensus.cc:1261] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.494066 9254 consensus_queue.cc:1035] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.499938 8873 raft_consensus.cc:2932] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.502388 2073 raft_consensus.cc:1261] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:04.502378 4964 raft_consensus.cc:1261] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:04.503068 8884 consensus_queue.cc:1035] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:04.503720 8884 consensus_queue.cc:1035] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:04.509603 1809 raft_consensus.cc:2932] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.516513 4565 raft_consensus.cc:1261] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:04.520757 8884 raft_consensus.cc:2932] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:04.521672 4964 raft_consensus.cc:2932] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:04.523583 1668 catalog_manager.cc:5484] T c4177c11467b49d49452ac10bd983627 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.526281 8625 consensus_queue.cc:1035] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:04.529181 2073 raft_consensus.cc:2932] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:04.530192 4813 tablet_service.cc:1514] Processing DeleteTablet for tablet c4177c11467b49d49452ac10bd983627 with delete_type TABLET_DATA_TOMBSTONED (TS f24648c92c384f1ba349eee801f985e2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33952
I20240503 08:13:04.536226 1668 catalog_manager.cc:5484] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:04.549082 4565 raft_consensus.cc:2932] T c4177c11467b49d49452ac10bd983627 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.550308 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c4177c11467b49d49452ac10bd983627 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:04.551463 9271 tablet_replica.cc:330] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:13:04.551852 9271 raft_consensus.cc:2229] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:04.552222 9271 raft_consensus.cc:2258] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:04.553293 9271 ts_tablet_manager.cc:1875] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:04.554194 4698 consensus_queue.cc:237] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:04.556890 2073 raft_consensus.cc:1261] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.557024 4964 raft_consensus.cc:1261] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:04.557734 8705 consensus_queue.cc:1035] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.558281 8705 consensus_queue.cc:1035] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:04.560897 8705 raft_consensus.cc:2932] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:04.564931 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ebd7877c8faf409a8d042bef38f218aa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:04.565579 4569 raft_consensus.cc:2932] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:04.566092 2073 raft_consensus.cc:2932] T ebd7877c8faf409a8d042bef38f218aa P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:04.569211 1670 catalog_manager.cc:5484] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:04.576320 9271 ts_tablet_manager.cc:1888] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:04.576495 9271 log.cc:1196] T c4177c11467b49d49452ac10bd983627 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/c4177c11467b49d49452ac10bd983627
I20240503 08:13:04.577533 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet c4177c11467b49d49452ac10bd983627 (table rebalancer_extra_table_28 [id=a4d2142b44bc4afa85d24f2a4c14b381]) successfully deleted
I20240503 08:13:04.578207 9173 tablet_copy_client.cc:308] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.198:46559
I20240503 08:13:04.579180 4723 tablet_copy_service.cc:140] P 31f41542452b4b1eb994ff4223d039fc: Received BeginTabletCopySession request for tablet 9c02a6bb7f7d4b42a6af774f342b0398 from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:41709)
I20240503 08:13:04.579407 4723 tablet_copy_service.cc:161] P 31f41542452b4b1eb994ff4223d039fc: Beginning new tablet copy session on tablet 9c02a6bb7f7d4b42a6af774f342b0398 from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:41709: session id = b8f581390da449ccb9e299011296405e-9c02a6bb7f7d4b42a6af774f342b0398
I20240503 08:13:04.580197 4548 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd7877c8faf409a8d042bef38f218aa with delete_type TABLET_DATA_TOMBSTONED (TS 5f95b05d111e4090bbc59dec394ed0ab not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39444
I20240503 08:13:04.582072 4723 tablet_copy_source_session.cc:205] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:04.582262 4967 raft_consensus.cc:2932] T ebd7877c8faf409a8d042bef38f218aa P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:04.583743 9173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c02a6bb7f7d4b42a6af774f342b0398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:04.592840 9173 tablet_copy_client.cc:783] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:04.593169 9173 tablet_copy_client.cc:647] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:04.595892 9173 tablet_copy_client.cc:523] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:04.598270 9274 tablet_replica.cc:330] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab: stopping tablet replica
I20240503 08:13:04.598682 9274 raft_consensus.cc:2229] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:04.600732 9173 tablet_bootstrap.cc:492] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:13:04.616556 9274 raft_consensus.cc:2258] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:04.618199 9173 tablet_bootstrap.cc:492] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:04.618664 9173 tablet_bootstrap.cc:492] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:13:04.618924 9173 ts_tablet_manager.cc:1367] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20240503 08:13:04.619691 9173 raft_consensus.cc:348] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:04.619973 9173 raft_consensus.cc:729] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:13:04.620239 9274 ts_tablet_manager.cc:1875] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:04.620190 9173 consensus_queue.cc:260] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:04.620752 9173 ts_tablet_manager.cc:1398] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:04.622103 4723 tablet_copy_service.cc:342] P 31f41542452b4b1eb994ff4223d039fc: Request end of tablet copy session b8f581390da449ccb9e299011296405e-9c02a6bb7f7d4b42a6af774f342b0398 received from {username='slave'} at 127.27.157.193:41709
I20240503 08:13:04.622236 4723 tablet_copy_service.cc:433] P 31f41542452b4b1eb994ff4223d039fc: ending tablet copy session b8f581390da449ccb9e299011296405e-9c02a6bb7f7d4b42a6af774f342b0398 on tablet 9c02a6bb7f7d4b42a6af774f342b0398 with peer b8f581390da449ccb9e299011296405e
I20240503 08:13:04.638509 9274 ts_tablet_manager.cc:1888] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:04.638690 9274 log.cc:1196] T ebd7877c8faf409a8d042bef38f218aa P 5f95b05d111e4090bbc59dec394ed0ab: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/wals/ebd7877c8faf409a8d042bef38f218aa
I20240503 08:13:04.639680 1656 catalog_manager.cc:4830] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): tablet ebd7877c8faf409a8d042bef38f218aa (table rebalancer_extra_table_16 [id=638cb29330ad4861bb96a9642c54c065]) successfully deleted
I20240503 08:13:04.668521 8895 raft_consensus.cc:1050] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab to VOTER
I20240503 08:13:04.669481 8895 consensus_queue.cc:237] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:13:04.686780 2201 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:04.687160 4569 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab [term 2 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:04.688853 8874 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:04.689503 8874 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:04.690366 1809 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 08:13:04.691118 8874 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:04.704330 9226 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:13:04.705817 1809 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:13:04.712352 2201 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:13:04.714687 1663 catalog_manager.cc:5484] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 8 to 9, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:04.724324 4967 consensus_queue.cc:237] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:13:04.725411 4569 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:13:04.731850 4565 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:04.733669 1809 raft_consensus.cc:1261] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 08:13:04.735987 8247 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:04.736580 8247 consensus_queue.cc:1035] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 08:13:04.739259 8247 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:13:04.743183 4565 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P 5f95b05d111e4090bbc59dec394ed0ab [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:13:04.744246 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 733bed28775e4779b74e92d8a395234f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:04.747179 1667 catalog_manager.cc:5484] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 9 to 10, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:04.753021 1807 raft_consensus.cc:2932] T 733bed28775e4779b74e92d8a395234f P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } }
I20240503 08:13:04.757429 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 733bed28775e4779b74e92d8a395234f with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42570
I20240503 08:13:04.758682 9253 tablet_replica.cc:330] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:13:04.758908 9253 raft_consensus.cc:2229] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:04.759135 9253 raft_consensus.cc:2258] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:04.760211 9253 ts_tablet_manager.cc:1875] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:04.769227 9253 ts_tablet_manager.cc:1888] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 08:13:04.769470 9253 log.cc:1196] T 733bed28775e4779b74e92d8a395234f P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/733bed28775e4779b74e92d8a395234f
I20240503 08:13:04.770290 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 733bed28775e4779b74e92d8a395234f (table rebalance_test_table_7 [id=ab330af65d474e5b9c89cd53a7f6a8b2]) successfully deleted
I20240503 08:13:04.800047 8923 raft_consensus.cc:1050] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:13:04.800922 8923 consensus_queue.cc:237] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:13:04.804327 4698 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:04.805141 9238 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:04.807734 7964 raft_consensus.cc:1050] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:13:04.808454 7964 consensus_queue.cc:237] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:04.811830 4834 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:04.812405 9238 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:04.815631 2201 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:04.818305 8923 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:04.822875 4569 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:04.822875 1807 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:04.823904 9238 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:04.824250 9307 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:04.829905 2074 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 08:13:04.831151 9308 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:04.833293 7964 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:04.839080 4698 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:04.839740 9238 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.844727 1807 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.850651 1663 catalog_manager.cc:5484] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:04.854815 1667 catalog_manager.cc:5484] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 7 to 8, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.871716 4833 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:04.876768 2201 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:04.879225 1943 consensus_queue.cc:237] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:13:04.882146 4699 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:04.883083 9308 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:04.887549 2073 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.888026 4834 raft_consensus.cc:1261] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:04.892136 7964 consensus_queue.cc:1035] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:04.907019 7964 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:04.912317 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c37418ac97343d1931a8e1239218a38 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:04.913888 4698 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:04.916821 4834 raft_consensus.cc:2932] T 7c37418ac97343d1931a8e1239218a38 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:04.921486 1667 catalog_manager.cc:5484] T 7c37418ac97343d1931a8e1239218a38 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 8 to 9, VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { promote: false } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:04.926049 2183 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c37418ac97343d1931a8e1239218a38 with delete_type TABLET_DATA_TOMBSTONED (TS 184c543cb05d49cc98ba1c56617be7a5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42570
I20240503 08:13:04.926482 9253 tablet_replica.cc:330] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5: stopping tablet replica
I20240503 08:13:04.926712 9253 raft_consensus.cc:2229] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:04.926926 9253 raft_consensus.cc:2258] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:04.928160 9253 ts_tablet_manager.cc:1875] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:04.928776 4569 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:04.938282 9253 ts_tablet_manager.cc:1888] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:04.938438 9253 log.cc:1196] T 7c37418ac97343d1931a8e1239218a38 P 184c543cb05d49cc98ba1c56617be7a5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/wals/7c37418ac97343d1931a8e1239218a38
I20240503 08:13:04.940985 1657 catalog_manager.cc:4830] TS 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): tablet 7c37418ac97343d1931a8e1239218a38 (table rebalance_test_table_9 [id=1d48efe8d72a4dfebb62a9559e3f4222]) successfully deleted
I20240503 08:13:04.948426 1943 consensus_queue.cc:237] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:04.951445 1807 raft_consensus.cc:1261] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:04.953965 1878 consensus_peers.cc:485] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d69a5126dff43af8ba3acc931d17799. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:04.953900 7964 consensus_queue.cc:1035] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:04.954982 4967 raft_consensus.cc:1261] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:04.955701 7964 consensus_queue.cc:1035] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:04.959439 7964 raft_consensus.cc:2932] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:04.960871 1807 raft_consensus.cc:2932] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:04.962288 4967 raft_consensus.cc:2932] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:04.969640 1663 catalog_manager.cc:5484] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:05.008598 4965 consensus_queue.cc:237] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:05.011436 1943 raft_consensus.cc:1261] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:05.011473 4834 raft_consensus.cc:1261] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:05.012130 8247 consensus_queue.cc:1035] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:05.012537 4902 consensus_peers.cc:485] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 415a75c852814ea89b65d294d682c571. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:05.012802 8247 consensus_queue.cc:1035] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:05.019361 8895 raft_consensus.cc:2932] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:05.020180 1943 raft_consensus.cc:2932] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:05.020941 4834 raft_consensus.cc:2932] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:05.028790 1667 catalog_manager.cc:5484] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:05.039649 1807 raft_consensus.cc:1203] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:05.433954 9392 ts_tablet_manager.cc:902] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:13:05.435395 9392 tablet_copy_client.cc:308] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:13:05.436208 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet 415a75c852814ea89b65d294d682c571 from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:35539)
I20240503 08:13:05.436398 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet 415a75c852814ea89b65d294d682c571 from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:35539: session id = b8f581390da449ccb9e299011296405e-415a75c852814ea89b65d294d682c571
I20240503 08:13:05.438895 4988 tablet_copy_source_session.cc:205] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:05.440239 9392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415a75c852814ea89b65d294d682c571. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.445665 9392 tablet_copy_client.cc:783] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:05.445911 9392 tablet_copy_client.cc:647] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:05.447618 9392 tablet_copy_client.cc:523] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:05.450182 9392 tablet_bootstrap.cc:492] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:13:05.455572 9392 tablet_bootstrap.cc:492] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:05.455962 9392 tablet_bootstrap.cc:492] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:13:05.456208 9392 ts_tablet_manager.cc:1367] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.005s
I20240503 08:13:05.456900 9392 raft_consensus.cc:348] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:05.457140 9392 raft_consensus.cc:729] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:13:05.457971 9392 consensus_queue.cc:260] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:05.458626 9392 ts_tablet_manager.cc:1398] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:13:05.459954 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session b8f581390da449ccb9e299011296405e-415a75c852814ea89b65d294d682c571 received from {username='slave'} at 127.27.157.193:35539
I20240503 08:13:05.463529 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session b8f581390da449ccb9e299011296405e-415a75c852814ea89b65d294d682c571 on tablet 415a75c852814ea89b65d294d682c571 with peer b8f581390da449ccb9e299011296405e
I20240503 08:13:05.472779 1943 consensus_queue.cc:237] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:05.475142 4569 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:05.475798 9307 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:05.476959 1807 raft_consensus.cc:1261] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 08:13:05.477946 9307 consensus_queue.cc:1035] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 08:13:05.478969 7964 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:05.479806 4569 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:05.481371 1807 raft_consensus.cc:2932] T 674e558bf5db4f169f8cdcb08c40f5fb P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:05.486912 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 674e558bf5db4f169f8cdcb08c40f5fb with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:05.488951 1670 catalog_manager.cc:5484] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 8 to 9, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:05.496260 9401 ts_tablet_manager.cc:902] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:13:05.498804 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 674e558bf5db4f169f8cdcb08c40f5fb with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35210
I20240503 08:13:05.501158 9406 tablet_replica.cc:330] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:13:05.501497 9406 raft_consensus.cc:2229] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:05.501911 9406 raft_consensus.cc:2258] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:05.503012 9406 ts_tablet_manager.cc:1875] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:05.504727 9401 tablet_copy_client.cc:308] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.194:41975
I20240503 08:13:05.505471 1964 tablet_copy_service.cc:140] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Received BeginTabletCopySession request for tablet 6d69a5126dff43af8ba3acc931d17799 from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:35673)
I20240503 08:13:05.505635 1964 tablet_copy_service.cc:161] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Beginning new tablet copy session on tablet 6d69a5126dff43af8ba3acc931d17799 from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:35673: session id = 184c543cb05d49cc98ba1c56617be7a5-6d69a5126dff43af8ba3acc931d17799
I20240503 08:13:05.508891 1964 tablet_copy_source_session.cc:205] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:05.512396 9401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d69a5126dff43af8ba3acc931d17799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:05.514210 9406 ts_tablet_manager.cc:1888] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 08:13:05.514350 9406 log.cc:1196] T 674e558bf5db4f169f8cdcb08c40f5fb P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/674e558bf5db4f169f8cdcb08c40f5fb
I20240503 08:13:05.515257 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet 674e558bf5db4f169f8cdcb08c40f5fb (table rebalance_test_table_17 [id=10478ffebe664fcda896887c04a61afa]) successfully deleted
I20240503 08:13:05.518955 9401 tablet_copy_client.cc:783] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:05.519217 9401 tablet_copy_client.cc:647] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:05.522125 9401 tablet_copy_client.cc:523] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:05.525566 9401 tablet_bootstrap.cc:492] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:13:05.543994 9401 tablet_bootstrap.cc:492] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:05.544382 9401 tablet_bootstrap.cc:492] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:13:05.544669 9401 ts_tablet_manager.cc:1367] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.001s
I20240503 08:13:05.545411 9401 raft_consensus.cc:348] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:05.545639 9401 raft_consensus.cc:729] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:13:05.545809 9401 consensus_queue.cc:260] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:05.546329 9401 ts_tablet_manager.cc:1398] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:05.552693 1964 tablet_copy_service.cc:342] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-6d69a5126dff43af8ba3acc931d17799 received from {username='slave'} at 127.27.157.196:35673
I20240503 08:13:05.552827 1964 tablet_copy_service.cc:433] P 315f3f0bcbf447a4b93b1aafcfbf0f5a: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-6d69a5126dff43af8ba3acc931d17799 on tablet 6d69a5126dff43af8ba3acc931d17799 with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:13:05.594682 8884 raft_consensus.cc:1050] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:13:05.595693 8884 consensus_queue.cc:237] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:05.599946 1807 raft_consensus.cc:1261] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:05.600797 9269 consensus_queue.cc:1035] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:05.604406 4833 raft_consensus.cc:1261] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:05.605022 9269 consensus_queue.cc:1035] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:05.610193 4833 raft_consensus.cc:2932] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:05.609561 9445 raft_consensus.cc:2932] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:05.618074 1663 catalog_manager.cc:5484] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:05.628743 1943 raft_consensus.cc:1261] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:05.629160 1807 raft_consensus.cc:2932] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:05.629464 9441 consensus_queue.cc:1035] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:05.649268 1943 raft_consensus.cc:2932] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:05.854426 2201 raft_consensus.cc:1203] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:06.060361 1807 raft_consensus.cc:1203] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:06.161455 4699 consensus_queue.cc:237] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:06.164603 1807 raft_consensus.cc:1261] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:06.164830 1942 raft_consensus.cc:1261] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:06.165236 9478 consensus_queue.cc:1035] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:06.165599 9269 consensus_queue.cc:1035] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:06.168315 9441 raft_consensus.cc:2932] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.169075 1807 raft_consensus.cc:2932] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.169086 1942 raft_consensus.cc:2932] T 9c02a6bb7f7d4b42a6af774f342b0398 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.173116 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9c02a6bb7f7d4b42a6af774f342b0398 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:06.177109 1663 catalog_manager.cc:5484] T 9c02a6bb7f7d4b42a6af774f342b0398 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.185976 4813 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c02a6bb7f7d4b42a6af774f342b0398 with delete_type TABLET_DATA_TOMBSTONED (TS f24648c92c384f1ba349eee801f985e2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33952
I20240503 08:13:06.186697 9482 tablet_replica.cc:330] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:13:06.187002 9482 raft_consensus.cc:2229] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:06.187356 9482 raft_consensus.cc:2258] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:06.188266 9482 ts_tablet_manager.cc:1875] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:06.193426 9482 ts_tablet_manager.cc:1888] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:06.193660 9482 log.cc:1196] T 9c02a6bb7f7d4b42a6af774f342b0398 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/9c02a6bb7f7d4b42a6af774f342b0398
I20240503 08:13:06.195752 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet 9c02a6bb7f7d4b42a6af774f342b0398 (table rebalancer_extra_table_14 [id=90ad7feb583d48b3ba66cbb595edbd6d]) successfully deleted
I20240503 08:13:06.311904 9307 raft_consensus.cc:1050] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:13:06.312806 9307 consensus_queue.cc:237] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:06.315433 2201 raft_consensus.cc:1261] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:06.315522 4965 raft_consensus.cc:1261] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:06.316160 1807 raft_consensus.cc:1261] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:06.316120 9238 consensus_queue.cc:1035] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:06.316380 9307 consensus_queue.cc:1035] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:06.317087 7964 consensus_queue.cc:1035] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:06.321130 7964 raft_consensus.cc:2932] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:06.321811 1807 raft_consensus.cc:2932] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:06.326413 2201 raft_consensus.cc:2932] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:06.329005 1668 catalog_manager.cc:5484] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:06.338152 4967 raft_consensus.cc:2932] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:06.353369 1942 consensus_queue.cc:237] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:06.355983 2201 raft_consensus.cc:1261] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:06.356643 4965 raft_consensus.cc:1261] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:06.356999 7964 consensus_queue.cc:1035] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:06.358214 7964 consensus_queue.cc:1035] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:06.361074 9238 raft_consensus.cc:2932] T 6d69a5126dff43af8ba3acc931d17799 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:06.361824 4967 raft_consensus.cc:2932] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:06.365352 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d69a5126dff43af8ba3acc931d17799 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:06.368711 1668 catalog_manager.cc:5484] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:06.369637 2201 raft_consensus.cc:2932] T 6d69a5126dff43af8ba3acc931d17799 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:06.375885 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d69a5126dff43af8ba3acc931d17799 with delete_type TABLET_DATA_TOMBSTONED (TS b8f581390da449ccb9e299011296405e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48256
I20240503 08:13:06.378247 9498 tablet_replica.cc:330] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:13:06.378769 9498 raft_consensus.cc:2229] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:06.379194 9498 raft_consensus.cc:2258] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:06.381239 9498 ts_tablet_manager.cc:1875] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:06.388561 9498 ts_tablet_manager.cc:1888] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:06.388712 9498 log.cc:1196] T 6d69a5126dff43af8ba3acc931d17799 P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/6d69a5126dff43af8ba3acc931d17799
I20240503 08:13:06.389871 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 6d69a5126dff43af8ba3acc931d17799 (table rebalancer_extra_table_32 [id=9bfbde099d3143e2b4a26a7c01253ea3]) successfully deleted
I20240503 08:13:06.606217 9509 raft_consensus.cc:1050] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:13:06.607179 9509 consensus_queue.cc:237] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:06.611819 1809 raft_consensus.cc:1261] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:06.622201 1942 raft_consensus.cc:1261] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:06.623948 9226 consensus_queue.cc:1035] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:06.624658 9226 consensus_queue.cc:1035] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:06.632356 4833 raft_consensus.cc:1261] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:06.633428 9509 consensus_queue.cc:1035] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:06.635419 9511 raft_consensus.cc:2932] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.636801 1942 raft_consensus.cc:2932] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.647301 1668 catalog_manager.cc:5484] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.649243 1807 raft_consensus.cc:2932] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.657454 4833 raft_consensus.cc:2932] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.665922 4965 consensus_queue.cc:237] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:06.669376 1807 raft_consensus.cc:1261] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:06.669396 1942 raft_consensus.cc:1261] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:06.670128 9226 consensus_queue.cc:1035] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:06.670485 8895 consensus_queue.cc:1035] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:06.674674 8895 raft_consensus.cc:2932] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.677891 1942 raft_consensus.cc:2932] T 415a75c852814ea89b65d294d682c571 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.679204 1809 raft_consensus.cc:2932] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.680945 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 415a75c852814ea89b65d294d682c571 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:06.689107 1668 catalog_manager.cc:5484] T 415a75c852814ea89b65d294d682c571 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:06.700923 4813 tablet_service.cc:1514] Processing DeleteTablet for tablet 415a75c852814ea89b65d294d682c571 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:33952
I20240503 08:13:06.721305 9519 tablet_replica.cc:330] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:13:06.721567 9519 raft_consensus.cc:2229] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:06.721814 9519 raft_consensus.cc:2258] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:06.722694 9519 ts_tablet_manager.cc:1875] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:06.729871 9519 ts_tablet_manager.cc:1888] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:06.730145 9519 log.cc:1196] T 415a75c852814ea89b65d294d682c571 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/415a75c852814ea89b65d294d682c571
I20240503 08:13:06.732729 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet 415a75c852814ea89b65d294d682c571 (table rebalancer_extra_table_22 [id=ad3e054abe66435cb6d8dec80bf19a25]) successfully deleted
I20240503 08:13:07.152320 1924 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 08:13:07.156913 4945 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 08:13:07.157410 2183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 08:13:07.228618 4683 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 08:13:07.281426 2054 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 08:13:07.316704 4548 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:07.340366 1793 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 08:13:07.378647 4813 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 08:13:07.542493 1807 consensus_queue.cc:237] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:07.545506 4965 raft_consensus.cc:1261] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.546120 1943 raft_consensus.cc:1261] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.546415 9318 consensus_queue.cc:1035] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.547367 9318 consensus_queue.cc:1035] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:07.547713 1747 consensus_peers.cc:485] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5785af2b9ba34a3b95c5ffdb024b1c8f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:07.550449 9489 raft_consensus.cc:2932] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:07.550984 4965 raft_consensus.cc:2932] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:07.557283 1663 catalog_manager.cc:5484] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:07.559007 1943 raft_consensus.cc:2932] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:07.585951 4700 consensus_queue.cc:237] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:07.588831 1807 raft_consensus.cc:1261] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:07.589453 4636 consensus_peers.cc:485] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f3e931766d44f0aaf902cf0e950f83b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:07.589620 4834 raft_consensus.cc:1261] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.589613 9441 consensus_queue.cc:1035] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.590657 9441 consensus_queue.cc:1035] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.593284 9441 raft_consensus.cc:2932] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:07.593726 1807 raft_consensus.cc:2932] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:07.609381 1669 catalog_manager.cc:5484] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:07.612876 4834 raft_consensus.cc:2932] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:07.648975 1942 consensus_queue.cc:237] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:13:07.652159 2204 raft_consensus.cc:1261] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:07.653220 1877 consensus_peers.cc:485] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a97625d919124c559b1e9a5cb785153f. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:07.653831 9238 consensus_queue.cc:1035] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.658640 4834 raft_consensus.cc:1261] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.659308 9238 consensus_queue.cc:1035] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.663434 7964 raft_consensus.cc:2932] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:13:07.679728 4834 raft_consensus.cc:2932] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:13:07.679838 2204 raft_consensus.cc:2932] T a97625d919124c559b1e9a5cb785153f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:13:07.688925 1663 catalog_manager.cc:5484] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:13:07.714694 4569 consensus_queue.cc:237] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
W20240503 08:13:07.717767 4503 consensus_peers.cc:485] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7e0415e028b4303bcf4ab64f4ec6b54. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:07.718480 2074 raft_consensus.cc:1261] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.718555 1942 raft_consensus.cc:1261] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer 5f95b05d111e4090bbc59dec394ed0ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.719210 9581 consensus_queue.cc:1035] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.719620 9319 consensus_queue.cc:1035] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.723074 9581 raft_consensus.cc:2932] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:07.723892 2074 raft_consensus.cc:2932] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:07.724483 1942 raft_consensus.cc:2932] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:07.731104 1669 catalog_manager.cc:5484] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "5f95b05d111e4090bbc59dec394ed0ab" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:07.748104 4834 consensus_queue.cc:237] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:07.751391 4700 raft_consensus.cc:1261] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.751623 2074 raft_consensus.cc:1261] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:07.751696 4768 consensus_peers.cc:485] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 826e7d6e6d2947fbb2a178e8e2a32d56. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:07.752094 9518 consensus_queue.cc:1035] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.752480 9241 consensus_queue.cc:1035] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.770902 9518 raft_consensus.cc:2932] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:07.771467 2074 raft_consensus.cc:2932] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:07.778367 1669 catalog_manager.cc:5484] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:07.791177 4700 raft_consensus.cc:2932] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:07.848323 2074 consensus_queue.cc:237] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:07.851119 4698 raft_consensus.cc:1261] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.851423 1807 raft_consensus.cc:1261] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.851985 9190 consensus_queue.cc:1035] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.852377 8881 consensus_queue.cc:1035] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:07.853277 2008 consensus_peers.cc:485] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c3070c262ed420a91ac0ed063d16471. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:07.857290 8881 raft_consensus.cc:2932] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:07.861424 1807 raft_consensus.cc:2932] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:07.871071 1669 catalog_manager.cc:5484] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:07.874948 4698 raft_consensus.cc:2932] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } } }
I20240503 08:13:07.911033 4965 consensus_queue.cc:237] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:13:07.916458 2074 raft_consensus.cc:1261] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.916522 4569 raft_consensus.cc:1261] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:07.916864 4902 consensus_peers.cc:485] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 438146d5fa194156b929d7190b11a5de. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:07.917209 9509 consensus_queue.cc:1035] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.922021 9509 consensus_queue.cc:1035] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.922508 9517 raft_consensus.cc:2932] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:13:07.924634 4569 raft_consensus.cc:2932] T 438146d5fa194156b929d7190b11a5de P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:13:07.932042 1663 catalog_manager.cc:5484] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:07.937799 2074 raft_consensus.cc:2932] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:13:07.954437 2074 consensus_queue.cc:237] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:13:07.957423 1809 raft_consensus.cc:1261] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:07.957535 4700 raft_consensus.cc:1261] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:07.957545 2010 consensus_peers.cc:485] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12b0c4a0b0564a279806d3fd6f981eed. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:07.958117 9638 consensus_queue.cc:1035] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.958750 9643 consensus_queue.cc:1035] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:07.961971 9638 raft_consensus.cc:2932] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:13:07.963078 1809 raft_consensus.cc:2932] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:13:07.963713 4699 raft_consensus.cc:2932] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } }
I20240503 08:13:07.968626 1663 catalog_manager.cc:5484] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 2, NON_VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:07.993085 2204 consensus_queue.cc:237] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:13:08.013556 4965 raft_consensus.cc:1261] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.014642 9599 consensus_queue.cc:1035] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:08.014777 4699 raft_consensus.cc:1261] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 184c543cb05d49cc98ba1c56617be7a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.015669 9599 consensus_queue.cc:1035] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:08.018358 2137 consensus_peers.cc:485] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ba03a430b4b498f9ba66d733d68b13c. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.018921 9599 raft_consensus.cc:2932] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:13:08.022794 4965 raft_consensus.cc:2932] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:13:08.031415 1663 catalog_manager.cc:5484] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New cstate: current_term: 1 leader_uuid: "184c543cb05d49cc98ba1c56617be7a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:13:08.038022 4699 raft_consensus.cc:2932] T 0ba03a430b4b498f9ba66d733d68b13c P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } } }
I20240503 08:13:08.053120 4699 consensus_queue.cc:237] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:08.056016 4569 raft_consensus.cc:1261] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.056113 4965 raft_consensus.cc:1261] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:08.056519 4636 consensus_peers.cc:485] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a792a309b3c43f482b4adb0fbcfadde. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.056892 9601 consensus_queue.cc:1035] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:08.058029 9601 consensus_queue.cc:1035] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:08.060909 9478 raft_consensus.cc:2932] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:08.061662 4965 raft_consensus.cc:2932] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:08.070021 1663 catalog_manager.cc:5484] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:08.098348 9681 ts_tablet_manager.cc:902] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:13:08.113509 4565 raft_consensus.cc:2932] T 7a792a309b3c43f482b4adb0fbcfadde P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } } }
I20240503 08:13:08.125701 9681 tablet_copy_client.cc:308] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.193:41181
I20240503 08:13:08.126437 1833 tablet_copy_service.cc:140] P b8f581390da449ccb9e299011296405e: Received BeginTabletCopySession request for tablet 5785af2b9ba34a3b95c5ffdb024b1c8f from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:40639)
I20240503 08:13:08.126614 1833 tablet_copy_service.cc:161] P b8f581390da449ccb9e299011296405e: Beginning new tablet copy session on tablet 5785af2b9ba34a3b95c5ffdb024b1c8f from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:40639: session id = 184c543cb05d49cc98ba1c56617be7a5-5785af2b9ba34a3b95c5ffdb024b1c8f
I20240503 08:13:08.132194 1833 tablet_copy_source_session.cc:205] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.133734 9681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5785af2b9ba34a3b95c5ffdb024b1c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.138607 9681 tablet_copy_client.cc:783] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.138839 9681 tablet_copy_client.cc:647] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.141080 9681 tablet_copy_client.cc:523] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.144291 9681 tablet_bootstrap.cc:492] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
I20240503 08:13:08.147073 4834 consensus_queue.cc:237] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:13:08.150631 4700 raft_consensus.cc:1261] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:08.151046 4769 consensus_peers.cc:485] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51f3edb7a72c4ba083429c150fa73363. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.151131 9681 tablet_bootstrap.cc:492] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.151532 9681 tablet_bootstrap.cc:492] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:13:08.151584 4565 raft_consensus.cc:1261] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.151777 9681 ts_tablet_manager.cc:1367] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:13:08.151948 9630 consensus_queue.cc:1035] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:08.152280 9518 consensus_queue.cc:1035] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:08.152554 9681 raft_consensus.cc:348] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:08.152777 9681 raft_consensus.cc:729] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:13:08.153012 9681 consensus_queue.cc:260] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:08.153717 9681 ts_tablet_manager.cc:1398] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:08.154472 1833 tablet_copy_service.cc:342] P b8f581390da449ccb9e299011296405e: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-5785af2b9ba34a3b95c5ffdb024b1c8f received from {username='slave'} at 127.27.157.196:40639
I20240503 08:13:08.154594 1833 tablet_copy_service.cc:433] P b8f581390da449ccb9e299011296405e: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-5785af2b9ba34a3b95c5ffdb024b1c8f on tablet 5785af2b9ba34a3b95c5ffdb024b1c8f with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:13:08.156692 9241 raft_consensus.cc:2932] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:13:08.165110 4565 raft_consensus.cc:2932] T 51f3edb7a72c4ba083429c150fa73363 P 5f95b05d111e4090bbc59dec394ed0ab [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:13:08.171408 1669 catalog_manager.cc:5484] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:08.189070 4699 raft_consensus.cc:2932] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } } }
I20240503 08:13:08.196731 9697 ts_tablet_manager.cc:902] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:08.206660 9704 ts_tablet_manager.cc:902] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a: Initiating tablet copy from peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:13:08.206894 9697 tablet_copy_client.cc:308] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.197:42465
I20240503 08:13:08.207856 4589 tablet_copy_service.cc:140] P 5f95b05d111e4090bbc59dec394ed0ab: Received BeginTabletCopySession request for tablet a7e0415e028b4303bcf4ab64f4ec6b54 from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:41097)
I20240503 08:13:08.208151 4589 tablet_copy_service.cc:161] P 5f95b05d111e4090bbc59dec394ed0ab: Beginning new tablet copy session on tablet a7e0415e028b4303bcf4ab64f4ec6b54 from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:41097: session id = b8f581390da449ccb9e299011296405e-a7e0415e028b4303bcf4ab64f4ec6b54
I20240503 08:13:08.209235 9704 tablet_copy_client.cc:308] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.198:46559
I20240503 08:13:08.210970 4589 tablet_copy_source_session.cc:205] T a7e0415e028b4303bcf4ab64f4ec6b54 P 5f95b05d111e4090bbc59dec394ed0ab: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.211303 4723 tablet_copy_service.cc:140] P 31f41542452b4b1eb994ff4223d039fc: Received BeginTabletCopySession request for tablet 8f3e931766d44f0aaf902cf0e950f83b from peer 43e2e01cdd314f9c872c4336ef74250a ({username='slave'} at 127.27.157.195:37471)
I20240503 08:13:08.211555 4723 tablet_copy_service.cc:161] P 31f41542452b4b1eb994ff4223d039fc: Beginning new tablet copy session on tablet 8f3e931766d44f0aaf902cf0e950f83b from peer 43e2e01cdd314f9c872c4336ef74250a at {username='slave'} at 127.27.157.195:37471: session id = 43e2e01cdd314f9c872c4336ef74250a-8f3e931766d44f0aaf902cf0e950f83b
I20240503 08:13:08.214110 4723 tablet_copy_source_session.cc:205] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.215781 9704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3e931766d44f0aaf902cf0e950f83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.215955 9697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e0415e028b4303bcf4ab64f4ec6b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.220423 4700 consensus_queue.cc:237] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:08.221014 9704 tablet_copy_client.cc:783] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.221326 9704 tablet_copy_client.cc:647] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.222138 9697 tablet_copy_client.cc:783] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.222398 9697 tablet_copy_client.cc:647] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.225044 9704 tablet_copy_client.cc:523] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.225390 9697 tablet_copy_client.cc:523] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.226701 4834 raft_consensus.cc:1261] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:08.227056 4636 consensus_peers.cc:485] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e613a6cbac2c4ad9b71442baa71ba9a3. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.227366 9679 consensus_queue.cc:1035] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:08.230058 9697 tablet_bootstrap.cc:492] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:13:08.232472 2074 raft_consensus.cc:1261] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.233297 9478 consensus_queue.cc:1035] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:08.263867 9680 raft_consensus.cc:2932] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:08.264693 4834 raft_consensus.cc:2932] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:08.272982 1663 catalog_manager.cc:5484] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:08.275290 2073 raft_consensus.cc:2932] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:08.275352 9704 tablet_bootstrap.cc:492] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:13:08.296176 9712 ts_tablet_manager.cc:902] T a97625d919124c559b1e9a5cb785153f P 5f95b05d111e4090bbc59dec394ed0ab: Initiating tablet copy from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:13:08.299834 9715 ts_tablet_manager.cc:902] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
F20240503 08:13:08.299738 9712 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-3219159f92774adbb7ecaa77ac267021.queue_time_us, tablet-download-pool-3219159f92774adbb7ecaa77ac267021.queue_time_us) (tablet-download-pool-3219159f92774adbb7ecaa77ac267021.run_cpu_time_us, tablet-download-pool-3219159f92774adbb7ecaa77ac267021.run_cpu_time_us) (tablet-download-pool-3219159f92774adbb7ecaa77ac267021.run_wall_time_us, tablet-download-pool-3219159f92774adbb7ecaa77ac267021.run_wall_time_us) (tablet-download-pool-48f411ec0bfc4a318b661441c49bd6ec.queue_time_us, tablet-download-pool-48f411ec0bfc4a318b661441c49bd6ec.queue_time_us) (tablet-download-pool-48f411ec0bfc4a318b661441c49bd6ec.run_cpu_time_us, tablet-download-pool-48f411ec0bfc4a318b661441c49bd6ec.run_cpu_time_us) (tablet-download-pool-48f411ec0bfc4a318b661441c49bd6ec.run_wall_time_us, tablet-download-pool-48f411ec0bfc4a318b661441c49bd6ec.run_wall_time_us) (tablet-download-pool-53197deeb8d7439c90b019a6a90857e0.queue_time_us, tablet-download-pool-53197deeb8d7439c90b019a6a90857e0.queue_time_us) (tablet-download-pool-53197deeb8d7439c90b019a6a90857e0.run_cpu_time_us, tablet-download-pool-53197deeb8d7439c90b019a6a90857e0.run_cpu_time_us) (tablet-download-pool-53197deeb8d7439c90b019a6a90857e0.run_wall_time_us, tablet-download-pool-53197deeb8d7439c90b019a6a90857e0.run_wall_time_us) (tablet-download-pool-674e558bf5db4f169f8cdcb08c40f5fb.queue_time_us, tablet-download-pool-674e558bf5db4f169f8cdcb08c40f5fb.queue_time_us) (tablet-download-pool-674e558bf5db4f169f8cdcb08c40f5fb.run_cpu_time_us, tablet-download-pool-674e558bf5db4f169f8cdcb08c40f5fb.run_cpu_time_us) (tablet-download-pool-674e558bf5db4f169f8cdcb08c40f5fb.run_wall_time_us, tablet-download-pool-674e558bf5db4f169f8cdcb08c40f5fb.run_wall_time_us) (tablet-download-pool-733bed28775e4779b74e92d8a395234f.queue_time_us, tablet-download-pool-733bed28775e4779b74e92d8a395234f.queue_time_us) (tablet-download-pool-733bed28775e4779b74e92d8a395234f.run_cpu_time_us, tablet-download-pool-733bed28775e4779b74e92d8a395234f.run_cpu_time_us) (tablet-download-pool-733bed28775e4779b74e92d8a395234f.run_wall_time_us, tablet-download-pool-733bed28775e4779b74e92d8a395234f.run_wall_time_us) (tablet-download-pool-79579b65538d47f7973fa51f973b3820.queue_time_us, tablet-download-pool-79579b65538d47f7973fa51f973b3820.queue_time_us) (tablet-download-pool-79579b65538d47f7973fa51f973b3820.run_cpu_time_us, tablet-download-pool-79579b65538d47f7973fa51f973b3820.run_cpu_time_us) (tablet-download-pool-79579b65538d47f7973fa51f973b3820.run_wall_time_us, tablet-download-pool-79579b65538d47f7973fa51f973b3820.run_wall_time_us) (tablet-download-pool-8737ef0936c641ccaf0b804b1d9c3e07.queue_time_us, tablet-download-pool-8737ef0936c641ccaf0b804b1d9c3e07.queue_time_us) (tablet-download-pool-8737ef0936c641ccaf0b804b1d9c3e07.run_cpu_time_us, tablet-download-pool-8737ef0936c641ccaf0b804b1d9c3e07.run_cpu_time_us) (tablet-download-pool-8737ef0936c641ccaf0b804b1d9c3e07.run_wall_time_us, tablet-download-pool-8737ef0936c641ccaf0b804b1d9c3e07.run_wall_time_us) (tablet-download-pool-89d9b1b3c23241d49bfcb914a5bd1777.queue_time_us, tablet-download-pool-89d9b1b3c23241d49bfcb914a5bd1777.queue_time_us) (tablet-download-pool-89d9b1b3c23241d49bfcb914a5bd1777.run_cpu_time_us, tablet-download-pool-89d9b1b3c23241d49bfcb914a5bd1777.run_cpu_time_us) (tablet-download-pool-89d9b1b3c23241d49bfcb914a5bd1777.run_wall_time_us, tablet-download-pool-89d9b1b3c23241d49bfcb914a5bd1777.run_wall_time_us) (tablet-download-pool-8fc8279b96f74e879b499b0143502c8f.queue_time_us, tablet-download-pool-8fc8279b96f74e879b499b0143502c8f.queue_time_us) (tablet-download-pool-8fc8279b96f74e879b499b0143502c8f.run_cpu_time_us, tablet-download-pool-8fc8279b96f74e879b499b0143502c8f.run_cpu_time_us) (tablet-download-pool-8fc8279b96f74e879b499b0143502c8f.run_wall_time_us, tablet-download-pool-8fc8279b96f74e879b499b0143502c8f.run_wall_time_us) (tablet-download-pool-9cb8a87410b047548efdd973bbc86714.queue_time_us, tablet-download-pool-9cb8a87410b047548efdd973bbc86714.queue_time_us) (tablet-download-pool-9cb8a87410b047548efdd973bbc86714.run_cpu_time_us, tablet-download-pool-9cb8a87410b047548efdd973bbc86714.run_cpu_time_us) (tablet-download-pool-9cb8a87410b047548efdd973bbc86714.run_wall_time_us, tablet-download-pool-9cb8a87410b047548efdd973bbc86714.run_wall_time_us) (tablet-download-pool-9e371231b34f45dfac455bbbbe156263.queue_time_us, tablet-download-pool-9e371231b34f45dfac455bbbbe156263.queue_time_us) (tablet-download-pool-9e371231b34f45dfac455bbbbe156263.run_cpu_time_us, tablet-download-pool-9e371231b34f45dfac455bbbbe156263.run_cpu_time_us) (tablet-download-pool-9e371231b34f45dfac455bbbbe156263.run_wall_time_us, tablet-download-pool-9e371231b34f45dfac455bbbbe156263.run_wall_time_us) (tablet-download-pool-a97625d919124c559b1e9a5cb785153f.queue_time_us, tablet-download-pool-a97625d919124c559b1e9a5cb785153f.queue_time_us) (tablet-download-pool-f4900ffdbc624376b32f054efdb100b5.queue_time_us, tablet-download-pool-f4900ffdbc624376b32f054efdb100b5.queue_time_us) (tablet-download-pool-f4900ffdbc624376b32f054efdb100b5.run_cpu_time_us, tablet-download-pool-f4900ffdbc624376b32f054efdb100b5.run_cpu_time_us) (tablet-download-pool-f4900ffdbc624376b32f054efdb100b5.run_wall_time_us, tablet-download-pool-f4900ffdbc624376b32f054efdb100b5.run_wall_time_us) (tablet-download-pool-f6cff57835b846e6b48c08bef52f5b68.queue_time_us, tablet-download-pool-f6cff57835b846e6b48c08bef52f5b68.queue_time_us) (tablet-download-pool-f6cff57835b846e6b48c08bef52f5b68.run_cpu_time_us, tablet-download-pool-f6cff57835b846e6b48c08bef52f5b68.run_cpu_time_us) (tablet-download-pool-f6cff57835b846e6b48c08bef52f5b68.run_wall_time_us, tablet-download-pool-f6cff57835b846e6b48c08bef52f5b68.run_wall_time_us) (tablet-download-pool-fba1e1952da44c0e956be1ef7f592f08.queue_time_us, tablet-download-pool-fba1e1952da44c0e956be1ef7f592f08.queue_time_us) (tablet-download-pool-fba1e1952da44c0e956be1ef7f592f08.run_cpu_time_us, tablet-download-pool-fba1e1952da44c0e956be1ef7f592f08.run_cpu_time_us) (tablet-download-pool-fba1e1952da44c0e956be1ef7f592f08.run_wall_time_us, tablet-download-pool-fba1e1952da44c0e956be1ef7f592f08.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f1e638d3dcd google::LogMessage::Fail() at ??:0
I20240503 08:13:08.314839 9697 tablet_bootstrap.cc:492] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.315634 9697 tablet_bootstrap.cc:492] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:13:08.315991 9697 ts_tablet_manager.cc:1367] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.086s user 0.023s sys 0.041s
@ 0x7f1e638d7b93 google::LogMessage::SendToLog() at ??:0
I20240503 08:13:08.317096 9697 raft_consensus.cc:348] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:08.317421 9697 raft_consensus.cc:729] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:13:08.317683 9697 consensus_queue.cc:260] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:08.318509 9697 ts_tablet_manager.cc:1398] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 08:13:08.325208 4589 tablet_copy_service.cc:342] P 5f95b05d111e4090bbc59dec394ed0ab: Request end of tablet copy session b8f581390da449ccb9e299011296405e-a7e0415e028b4303bcf4ab64f4ec6b54 received from {username='slave'} at 127.27.157.193:41097
I20240503 08:13:08.325361 4589 tablet_copy_service.cc:433] P 5f95b05d111e4090bbc59dec394ed0ab: ending tablet copy session b8f581390da449ccb9e299011296405e-a7e0415e028b4303bcf4ab64f4ec6b54 on tablet a7e0415e028b4303bcf4ab64f4ec6b54 with peer b8f581390da449ccb9e299011296405e
@ 0x7f1e638d37cc google::LogMessage::Flush() at ??:0
I20240503 08:13:08.332943 9715 tablet_copy_client.cc:308] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:13:08.333693 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet 826e7d6e6d2947fbb2a178e8e2a32d56 from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:52427)
I20240503 08:13:08.333910 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet 826e7d6e6d2947fbb2a178e8e2a32d56 from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:52427: session id = 00b61625d8a7460892c118a2570afda5-826e7d6e6d2947fbb2a178e8e2a32d56
I20240503 08:13:08.336791 9704 tablet_bootstrap.cc:492] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.337170 9704 tablet_bootstrap.cc:492] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap complete.
I20240503 08:13:08.337400 9704 ts_tablet_manager.cc:1367] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.000s
I20240503 08:13:08.337409 4855 tablet_copy_source_session.cc:205] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.342854 4833 consensus_queue.cc:237] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:08.345252 9681 ts_tablet_manager.cc:902] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:13:08.349401 9715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 826e7d6e6d2947fbb2a178e8e2a32d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.338250 9704 raft_consensus.cc:348] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:08.353899 9704 raft_consensus.cc:729] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: LEARNER
I20240503 08:13:08.354162 9704 consensus_queue.cc:260] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:08.354898 2073 raft_consensus.cc:1261] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.355000 2204 raft_consensus.cc:1261] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.356284 9630 consensus_queue.cc:1035] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:08.356719 9704 ts_tablet_manager.cc:1398] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
W20240503 08:13:08.358076 4769 consensus_peers.cc:485] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a03edb25e08e495c8dcb46adc9d8d2a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.359119 4723 tablet_copy_service.cc:342] P 31f41542452b4b1eb994ff4223d039fc: Request end of tablet copy session 43e2e01cdd314f9c872c4336ef74250a-8f3e931766d44f0aaf902cf0e950f83b received from {username='slave'} at 127.27.157.195:37471
I20240503 08:13:08.359279 4723 tablet_copy_service.cc:433] P 31f41542452b4b1eb994ff4223d039fc: ending tablet copy session 43e2e01cdd314f9c872c4336ef74250a-8f3e931766d44f0aaf902cf0e950f83b on tablet 8f3e931766d44f0aaf902cf0e950f83b with peer 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:13:08.360517 9630 consensus_queue.cc:1035] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
@ 0x7f1e638d4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 08:13:08.362468 9715 tablet_copy_client.cc:783] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.362794 9715 tablet_copy_client.cc:647] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.365036 9715 tablet_copy_client.cc:523] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.372422 9715 tablet_bootstrap.cc:492] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
@ 0x7f1e66614742 kudu::TraceMetrics::InternName() at ??:0
I20240503 08:13:08.380273 9733 raft_consensus.cc:2932] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:08.382833 2201 raft_consensus.cc:2932] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:08.392817 9715 tablet_bootstrap.cc:492] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.393028 1663 catalog_manager.cc:5484] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:08.393496 9715 tablet_bootstrap.cc:492] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:13:08.393774 9715 ts_tablet_manager.cc:1367] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.002s
I20240503 08:13:08.393499 2074 raft_consensus.cc:2932] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:08.394618 9715 raft_consensus.cc:348] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:08.394846 9715 raft_consensus.cc:729] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:13:08.395068 9715 consensus_queue.cc:260] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
@ 0x7f1e665e0f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240503 08:13:08.397238 9715 ts_tablet_manager.cc:1398] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:13:08.398880 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-826e7d6e6d2947fbb2a178e8e2a32d56 received from {username='slave'} at 127.27.157.200:52427
I20240503 08:13:08.399013 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 00b61625d8a7460892c118a2570afda5-826e7d6e6d2947fbb2a178e8e2a32d56 on tablet 826e7d6e6d2947fbb2a178e8e2a32d56 with peer 00b61625d8a7460892c118a2570afda5
@ 0x7f1e665dacd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f1e75a2e8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 08:13:08.437381 9681 tablet_copy_client.cc:308] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:13:08.438237 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 0c3070c262ed420a91ac0ed063d16471 from peer 184c543cb05d49cc98ba1c56617be7a5 ({username='slave'} at 127.27.157.196:51463)
I20240503 08:13:08.438419 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 0c3070c262ed420a91ac0ed063d16471 from peer 184c543cb05d49cc98ba1c56617be7a5 at {username='slave'} at 127.27.157.196:51463: session id = 184c543cb05d49cc98ba1c56617be7a5-0c3070c262ed420a91ac0ed063d16471
@ 0x7f1e75a46daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 08:13:08.442293 2093 tablet_copy_source_session.cc:205] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.445560 9681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3070c262ed420a91ac0ed063d16471. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.456288 9732 ts_tablet_manager.cc:902] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Initiating tablet copy from peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:13:08.458014 9681 tablet_copy_client.cc:783] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.458302 9681 tablet_copy_client.cc:647] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Starting download of 1 WAL segments...
@ 0x7f1e75b71480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 08:13:08.461769 9681 tablet_copy_client.cc:523] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f1e75ba9a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 08:13:08.472142 9681 tablet_bootstrap.cc:492] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap starting.
@ 0x7f1e75b8c981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240503 08:13:08.476171 4834 consensus_queue.cc:237] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
@ 0x7f1e75b8c70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240503 08:13:08.481444 4700 raft_consensus.cc:1261] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 08:13:08.482210 4768 consensus_peers.cc:485] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 -> Peer 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): Couldn't send request to peer 00b61625d8a7460892c118a2570afda5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6b22a007a2540ea9364579ac680fcf4. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.482604 9518 consensus_queue.cc:1035] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:08.484306 9681 tablet_bootstrap.cc:492] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.484810 9681 tablet_bootstrap.cc:492] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5: Bootstrap complete.
I20240503 08:13:08.485080 9681 ts_tablet_manager.cc:1367] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
@ 0x7f1e665e4b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 08:13:08.485921 9681 raft_consensus.cc:348] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:08.486253 9681 raft_consensus.cc:729] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 184c543cb05d49cc98ba1c56617be7a5, State: Initialized, Role: LEARNER
I20240503 08:13:08.486272 1809 raft_consensus.cc:1261] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.486584 9681 consensus_queue.cc:260] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: NON_VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: true } }
I20240503 08:13:08.487053 9518 consensus_queue.cc:1035] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:08.487656 9681 ts_tablet_manager.cc:1398] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
@ 0x7f1e665e819c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240503 08:13:08.489485 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session 184c543cb05d49cc98ba1c56617be7a5-0c3070c262ed420a91ac0ed063d16471 received from {username='slave'} at 127.27.157.196:51463
I20240503 08:13:08.489641 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session 184c543cb05d49cc98ba1c56617be7a5-0c3070c262ed420a91ac0ed063d16471 on tablet 0c3070c262ed420a91ac0ed063d16471 with peer 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:13:08.491145 9518 raft_consensus.cc:2932] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
@ 0x7f1e665e8019 std::_Function_handler<>::_M_invoke() at ??:0
I20240503 08:13:08.495687 1809 raft_consensus.cc:2932] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
I20240503 08:13:08.496084 4700 raft_consensus.cc:2932] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1e665c301e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1e699486db start_thread at ??:0
I20240503 08:13:08.507911 1663 catalog_manager.cc:5484] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index -1 to 2, NON_VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) added. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } } }
@ 0x7f1e6278671f clone at ??:0
I20240503 08:13:08.512357 9732 tablet_copy_client.cc:308] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.200:45283
I20240503 08:13:08.513512 4988 tablet_copy_service.cc:140] P 00b61625d8a7460892c118a2570afda5: Received BeginTabletCopySession request for tablet 438146d5fa194156b929d7190b11a5de from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a ({username='slave'} at 127.27.157.194:41645)
I20240503 08:13:08.513656 4988 tablet_copy_service.cc:161] P 00b61625d8a7460892c118a2570afda5: Beginning new tablet copy session on tablet 438146d5fa194156b929d7190b11a5de from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a at {username='slave'} at 127.27.157.194:41645: session id = 315f3f0bcbf447a4b93b1aafcfbf0f5a-438146d5fa194156b929d7190b11a5de
I20240503 08:13:08.516225 4988 tablet_copy_source_session.cc:205] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.521112 9732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438146d5fa194156b929d7190b11a5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.527441 9732 tablet_copy_client.cc:783] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.527746 9732 tablet_copy_client.cc:647] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.531352 9732 tablet_copy_client.cc:523] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.537784 9761 ts_tablet_manager.cc:902] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:13:08.544963 9732 tablet_bootstrap.cc:492] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:13:08.545320 9704 ts_tablet_manager.cc:902] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a: Initiating tablet copy from peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:13:08.561254 2201 raft_consensus.cc:1203] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:08.582437 9732 tablet_bootstrap.cc:492] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.582913 9732 tablet_bootstrap.cc:492] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap complete.
I20240503 08:13:08.583298 9732 ts_tablet_manager.cc:1367] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.007s
I20240503 08:13:08.584287 9732 raft_consensus.cc:348] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:13:08.584617 9732 raft_consensus.cc:729] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: LEARNER
I20240503 08:13:08.584825 9732 consensus_queue.cc:260] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:13:08.585448 9732 ts_tablet_manager.cc:1398] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:08.587764 4988 tablet_copy_service.cc:342] P 00b61625d8a7460892c118a2570afda5: Request end of tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-438146d5fa194156b929d7190b11a5de received from {username='slave'} at 127.27.157.194:41645
I20240503 08:13:08.587899 4988 tablet_copy_service.cc:433] P 00b61625d8a7460892c118a2570afda5: ending tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-438146d5fa194156b929d7190b11a5de on tablet 438146d5fa194156b929d7190b11a5de with peer 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:13:08.604455 9704 tablet_copy_client.cc:308] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.198:46559
I20240503 08:13:08.605257 4723 tablet_copy_service.cc:140] P 31f41542452b4b1eb994ff4223d039fc: Received BeginTabletCopySession request for tablet 7a792a309b3c43f482b4adb0fbcfadde from peer 43e2e01cdd314f9c872c4336ef74250a ({username='slave'} at 127.27.157.195:37471)
I20240503 08:13:08.605471 4723 tablet_copy_service.cc:161] P 31f41542452b4b1eb994ff4223d039fc: Beginning new tablet copy session on tablet 7a792a309b3c43f482b4adb0fbcfadde from peer 43e2e01cdd314f9c872c4336ef74250a at {username='slave'} at 127.27.157.195:37471: session id = 43e2e01cdd314f9c872c4336ef74250a-7a792a309b3c43f482b4adb0fbcfadde
I20240503 08:13:08.608436 4723 tablet_copy_source_session.cc:205] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.613128 9704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a792a309b3c43f482b4adb0fbcfadde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.623853 9704 tablet_copy_client.cc:783] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.624152 9704 tablet_copy_client.cc:647] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.629226 9704 tablet_copy_client.cc:523] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.629509 9775 ts_tablet_manager.cc:902] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:13:08.634316 9761 tablet_copy_client.cc:308] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:13:08.635373 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet 51f3edb7a72c4ba083429c150fa73363 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a ({username='slave'} at 127.27.157.194:55239)
I20240503 08:13:08.635565 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet 51f3edb7a72c4ba083429c150fa73363 from peer 315f3f0bcbf447a4b93b1aafcfbf0f5a at {username='slave'} at 127.27.157.194:55239: session id = 315f3f0bcbf447a4b93b1aafcfbf0f5a-51f3edb7a72c4ba083429c150fa73363
I20240503 08:13:08.638753 4855 tablet_copy_source_session.cc:205] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.640537 9761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f3edb7a72c4ba083429c150fa73363. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.675077 2073 raft_consensus.cc:1203] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:08.678558 9704 tablet_bootstrap.cc:492] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap starting.
I20240503 08:13:08.685274 9775 tablet_copy_client.cc:308] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:13:08.689529 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet 12b0c4a0b0564a279806d3fd6f981eed from peer f24648c92c384f1ba349eee801f985e2 ({username='slave'} at 127.27.157.199:52311)
I20240503 08:13:08.689708 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet 12b0c4a0b0564a279806d3fd6f981eed from peer f24648c92c384f1ba349eee801f985e2 at {username='slave'} at 127.27.157.199:52311: session id = f24648c92c384f1ba349eee801f985e2-12b0c4a0b0564a279806d3fd6f981eed
I20240503 08:13:08.689711 2073 consensus_queue.cc:237] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:08.692724 2093 tablet_copy_source_session.cc:205] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.696586 9775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12b0c4a0b0564a279806d3fd6f981eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.704710 9761 tablet_copy_client.cc:783] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.705022 9761 tablet_copy_client.cc:647] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.706930 9761 tablet_copy_client.cc:523] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 08:13:08.716351 2009 consensus_peers.cc:485] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2d33b2dc884448483072fa6ab825f0b. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:08.717078 4700 raft_consensus.cc:1261] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:13:08.717602 9643 consensus_queue.cc:1035] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:08.720841 9761 tablet_bootstrap.cc:492] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap starting.
I20240503 08:13:08.726585 9704 tablet_bootstrap.cc:492] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.727248 9704 tablet_bootstrap.cc:492] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a: Bootstrap complete.
I20240503 08:13:08.727891 9704 ts_tablet_manager.cc:1367] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.003s
I20240503 08:13:08.728933 9704 raft_consensus.cc:348] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:08.730594 9704 raft_consensus.cc:729] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Initialized, Role: LEARNER
I20240503 08:13:08.730824 9704 consensus_queue.cc:260] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: NON_VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: true } }
I20240503 08:13:08.731475 9704 ts_tablet_manager.cc:1398] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:08.731896 9775 tablet_copy_client.cc:783] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.732239 9775 tablet_copy_client.cc:647] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.734093 4723 tablet_copy_service.cc:342] P 31f41542452b4b1eb994ff4223d039fc: Request end of tablet copy session 43e2e01cdd314f9c872c4336ef74250a-7a792a309b3c43f482b4adb0fbcfadde received from {username='slave'} at 127.27.157.195:37471
I20240503 08:13:08.734172 9775 tablet_copy_client.cc:523] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.734210 4723 tablet_copy_service.cc:433] P 31f41542452b4b1eb994ff4223d039fc: ending tablet copy session 43e2e01cdd314f9c872c4336ef74250a-7a792a309b3c43f482b4adb0fbcfadde on tablet 7a792a309b3c43f482b4adb0fbcfadde with peer 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:13:08.737686 9775 tablet_bootstrap.cc:492] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2: Bootstrap starting.
I20240503 08:13:08.744691 9786 raft_consensus.cc:2932] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:08.745832 4700 raft_consensus.cc:2932] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } } }
I20240503 08:13:08.753485 9761 tablet_bootstrap.cc:492] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.754069 9761 tablet_bootstrap.cc:492] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Bootstrap complete.
I20240503 08:13:08.754384 9761 ts_tablet_manager.cc:1367] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.000s
I20240503 08:13:08.755390 9761 raft_consensus.cc:348] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:13:08.756944 9761 raft_consensus.cc:729] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Initialized, Role: LEARNER
I20240503 08:13:08.757205 9761 consensus_queue.cc:260] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: NON_VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: true } }
I20240503 08:13:08.762269 1667 catalog_manager.cc:5484] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index -1 to 2, NON_VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) added. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:08.763387 9761 ts_tablet_manager.cc:1398] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240503 08:13:08.768698 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-51f3edb7a72c4ba083429c150fa73363 received from {username='slave'} at 127.27.157.194:55239
I20240503 08:13:08.768932 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 315f3f0bcbf447a4b93b1aafcfbf0f5a-51f3edb7a72c4ba083429c150fa73363 on tablet 51f3edb7a72c4ba083429c150fa73363 with peer 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:13:08.772109 9715 ts_tablet_manager.cc:902] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:13:08.794127 9775 tablet_bootstrap.cc:492] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.794577 9775 tablet_bootstrap.cc:492] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2: Bootstrap complete.
I20240503 08:13:08.794813 9775 ts_tablet_manager.cc:1367] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2: Time spent bootstrapping tablet: real 0.057s user 0.016s sys 0.018s
I20240503 08:13:08.795601 9775 raft_consensus.cc:348] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:13:08.795821 9775 raft_consensus.cc:729] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Initialized, Role: LEARNER
I20240503 08:13:08.796026 9775 consensus_queue.cc:260] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } }
I20240503 08:13:08.797900 9775 ts_tablet_manager.cc:1398] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 08:13:08.798539 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session f24648c92c384f1ba349eee801f985e2-12b0c4a0b0564a279806d3fd6f981eed received from {username='slave'} at 127.27.157.199:52311
I20240503 08:13:08.798660 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session f24648c92c384f1ba349eee801f985e2-12b0c4a0b0564a279806d3fd6f981eed on tablet 12b0c4a0b0564a279806d3fd6f981eed with peer f24648c92c384f1ba349eee801f985e2
I20240503 08:13:08.812839 9697 ts_tablet_manager.cc:902] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:13:08.819578 1942 raft_consensus.cc:1203] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:08.832103 9715 tablet_copy_client.cc:308] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.198:46559
I20240503 08:13:08.832799 4723 tablet_copy_service.cc:140] P 31f41542452b4b1eb994ff4223d039fc: Received BeginTabletCopySession request for tablet e613a6cbac2c4ad9b71442baa71ba9a3 from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:50017)
I20240503 08:13:08.832954 4723 tablet_copy_service.cc:161] P 31f41542452b4b1eb994ff4223d039fc: Beginning new tablet copy session on tablet e613a6cbac2c4ad9b71442baa71ba9a3 from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:50017: session id = 00b61625d8a7460892c118a2570afda5-e613a6cbac2c4ad9b71442baa71ba9a3
I20240503 08:13:08.836107 4723 tablet_copy_source_session.cc:205] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.837566 4967 raft_consensus.cc:1203] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:08.838752 9715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e613a6cbac2c4ad9b71442baa71ba9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.857091 9715 tablet_copy_client.cc:783] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.857429 9715 tablet_copy_client.cc:647] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.864230 9715 tablet_copy_client.cc:523] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.878630 2201 raft_consensus.cc:1203] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:08.881778 9715 tablet_bootstrap.cc:492] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
I20240503 08:13:08.888768 9697 tablet_copy_client.cc:308] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:13:08.889793 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet a03edb25e08e495c8dcb46adc9d8d2a4 from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:54589)
I20240503 08:13:08.889982 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet a03edb25e08e495c8dcb46adc9d8d2a4 from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:54589: session id = b8f581390da449ccb9e299011296405e-a03edb25e08e495c8dcb46adc9d8d2a4
I20240503 08:13:08.892661 4855 tablet_copy_source_session.cc:205] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.894016 9697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a03edb25e08e495c8dcb46adc9d8d2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:08.906577 9697 tablet_copy_client.cc:783] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:08.906845 9697 tablet_copy_client.cc:647] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:08.913036 9697 tablet_copy_client.cc:523] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:08.917024 9697 tablet_bootstrap.cc:492] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:13:08.924505 9697 tablet_bootstrap.cc:492] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.924952 9697 tablet_bootstrap.cc:492] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:13:08.925248 9697 ts_tablet_manager.cc:1367] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240503 08:13:08.926059 9697 raft_consensus.cc:348] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:08.926349 9697 raft_consensus.cc:729] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:13:08.926578 9697 consensus_queue.cc:260] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:08.929420 9697 ts_tablet_manager.cc:1398] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20240503 08:13:08.930207 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session b8f581390da449ccb9e299011296405e-a03edb25e08e495c8dcb46adc9d8d2a4 received from {username='slave'} at 127.27.157.193:54589
I20240503 08:13:08.930342 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session b8f581390da449ccb9e299011296405e-a03edb25e08e495c8dcb46adc9d8d2a4 on tablet a03edb25e08e495c8dcb46adc9d8d2a4 with peer b8f581390da449ccb9e299011296405e
I20240503 08:13:08.938284 9715 tablet_bootstrap.cc:492] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:08.938799 9715 tablet_bootstrap.cc:492] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:13:08.939069 9715 ts_tablet_manager.cc:1367] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.004s
I20240503 08:13:08.940065 9808 ts_tablet_manager.cc:902] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5: Initiating tablet copy from peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:13:08.944579 9715 raft_consensus.cc:348] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:08.944823 9715 raft_consensus.cc:729] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
I20240503 08:13:08.945024 9715 consensus_queue.cc:260] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:08.945551 9715 ts_tablet_manager.cc:1398] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240503 08:13:08.947582 4723 tablet_copy_service.cc:342] P 31f41542452b4b1eb994ff4223d039fc: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-e613a6cbac2c4ad9b71442baa71ba9a3 received from {username='slave'} at 127.27.157.200:50017
I20240503 08:13:08.947723 4723 tablet_copy_service.cc:433] P 31f41542452b4b1eb994ff4223d039fc: ending tablet copy session 00b61625d8a7460892c118a2570afda5-e613a6cbac2c4ad9b71442baa71ba9a3 on tablet e613a6cbac2c4ad9b71442baa71ba9a3 with peer 00b61625d8a7460892c118a2570afda5
I20240503 08:13:08.983667 1943 raft_consensus.cc:1203] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:08.994939 9808 tablet_copy_client.cc:308] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.199:39455
I20240503 08:13:08.995889 4855 tablet_copy_service.cc:140] P f24648c92c384f1ba349eee801f985e2: Received BeginTabletCopySession request for tablet d6b22a007a2540ea9364579ac680fcf4 from peer 00b61625d8a7460892c118a2570afda5 ({username='slave'} at 127.27.157.200:52427)
I20240503 08:13:08.996023 4855 tablet_copy_service.cc:161] P f24648c92c384f1ba349eee801f985e2: Beginning new tablet copy session on tablet d6b22a007a2540ea9364579ac680fcf4 from peer 00b61625d8a7460892c118a2570afda5 at {username='slave'} at 127.27.157.200:52427: session id = 00b61625d8a7460892c118a2570afda5-d6b22a007a2540ea9364579ac680fcf4
I20240503 08:13:08.998081 4855 tablet_copy_source_session.cc:205] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:08.999580 9808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b22a007a2540ea9364579ac680fcf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.008637 9808 tablet_copy_client.cc:783] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:09.011627 9808 tablet_copy_client.cc:647] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:09.018833 9808 tablet_copy_client.cc:523] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:09.022312 9808 tablet_bootstrap.cc:492] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5: Bootstrap starting.
W20240503 08:13:09.027940 2137 connection.cc:535] client connection to 127.27.157.197:42465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.028241 2137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.029242 4767 connection.cc:535] client connection to 127.27.157.197:42465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.029532 4767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:09.034188 9808 tablet_bootstrap.cc:492] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240503 08:13:09.035550 2007 connection.cc:535] client connection to 127.27.157.197:42465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.035827 2007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:09.038316 9808 tablet_bootstrap.cc:492] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5: Bootstrap complete.
I20240503 08:13:09.038755 9808 ts_tablet_manager.cc:1367] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.002s
I20240503 08:13:09.040524 9808 raft_consensus.cc:348] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
I20240503 08:13:09.040817 9808 raft_consensus.cc:729] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Initialized, Role: LEARNER
W20240503 08:13:09.041304 4635 connection.cc:535] client connection to 127.27.157.197:42465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:09.041136 9808 consensus_queue.cc:260] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: NON_VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: true } }
W20240503 08:13:09.041603 4635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:09.042614 9808 ts_tablet_manager.cc:1398] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:09.044142 4855 tablet_copy_service.cc:342] P f24648c92c384f1ba349eee801f985e2: Request end of tablet copy session 00b61625d8a7460892c118a2570afda5-d6b22a007a2540ea9364579ac680fcf4 received from {username='slave'} at 127.27.157.200:52427
I20240503 08:13:09.044302 4855 tablet_copy_service.cc:433] P f24648c92c384f1ba349eee801f985e2: ending tablet copy session 00b61625d8a7460892c118a2570afda5-d6b22a007a2540ea9364579ac680fcf4 on tablet d6b22a007a2540ea9364579ac680fcf4 with peer 00b61625d8a7460892c118a2570afda5
W20240503 08:13:09.055274 4900 connection.cc:535] client connection to 127.27.157.197:42465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.055644 4900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.055991 1877 connection.cc:535] client connection to 127.27.157.197:42465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.056288 1877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.061182 1746 connection.cc:535] client connection to 127.27.157.197:42465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.061483 1746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.063024 9829 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:13:09.057662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.157.197:42465 (local address 127.27.157.194:57391)
0503 08:13:09.058142 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:09.058162 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:09.058307 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:09.062274 (+ 3967us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":2716,"threads_started":1}
W20240503 08:13:09.063185 1877 consensus_peers.cc:462] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.063293 1877 consensus_peers.cc:485] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.063422 1877 consensus_peers.cc:485] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.063490 1877 consensus_peers.cc:485] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.063563 1877 consensus_peers.cc:485] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.081058 4767 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.197:42465, user_credentials={real_user=slave}} blocked reactor thread for 51613.8us
W20240503 08:13:09.083375 4900 consensus_peers.cc:485] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.083631 4900 consensus_peers.cc:485] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.083741 4900 consensus_peers.cc:485] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.083814 4900 consensus_peers.cc:485] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.083896 4900 consensus_peers.cc:485] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.083985 4900 consensus_peers.cc:485] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.084084 4900 consensus_peers.cc:485] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.084173 4900 consensus_peers.cc:485] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.084287 4900 consensus_peers.cc:485] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.092386 9822 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:13:09.029515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.157.197:42465 (local address 127.27.157.196:53213)
0503 08:13:09.043077 (+ 13562us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:09.043099 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:09.043268 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:09.091022 (+ 47754us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":13450,"thread_start_us":13362,"threads_started":1}
W20240503 08:13:09.094633 1877 consensus_peers.cc:485] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.099009 9825 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:13:09.041834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.157.197:42465 (local address 127.27.157.195:48043)
0503 08:13:09.093684 (+ 51850us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:09.096827 (+ 3143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":51565,"thread_start_us":51440,"threads_started":1}
W20240503 08:13:09.099623 2007 consensus_peers.cc:485] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.100136 2007 consensus_peers.cc:485] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.100512 2007 consensus_peers.cc:485] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.100678 2007 consensus_peers.cc:485] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.100917 2007 consensus_peers.cc:485] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.101212 2137 consensus_peers.cc:485] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.101624 2137 consensus_peers.cc:485] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.101756 2137 consensus_peers.cc:462] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:09.105847 1746 consensus_peers.cc:485] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.106045 1746 consensus_peers.cc:485] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.106156 1746 consensus_peers.cc:485] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.107417 9769 raft_consensus.cc:1050] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
W20240503 08:13:09.108292 4635 consensus_peers.cc:485] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.108459 4635 consensus_peers.cc:485] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.108291 9769 consensus_queue.cc:237] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
W20240503 08:13:09.108582 4635 consensus_peers.cc:485] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.108659 9679 raft_consensus.cc:1050] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc: attempting to promote NON_VOTER 43e2e01cdd314f9c872c4336ef74250a to VOTER
W20240503 08:13:09.109086 4767 consensus_peers.cc:485] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.109236 4767 consensus_peers.cc:485] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.109354 4767 consensus_peers.cc:485] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:09.109447 4767 consensus_peers.cc:485] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.109562 9679 consensus_queue.cc:237] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:09.112056 2073 raft_consensus.cc:1261] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.117995 4834 raft_consensus.cc:1203] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:09.118988 9478 consensus_queue.cc:1035] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.119673 1807 raft_consensus.cc:1261] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.120306 9680 consensus_queue.cc:1035] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.122820 4834 raft_consensus.cc:1261] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.124354 9844 consensus_queue.cc:1035] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.128489 1943 raft_consensus.cc:1261] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.128530 2201 raft_consensus.cc:1261] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.129050 2074 raft_consensus.cc:1203] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:09.129166 9766 consensus_queue.cc:1035] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.131340 9846 consensus_queue.cc:1035] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.137588 4965 raft_consensus.cc:1261] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.138295 9769 consensus_queue.cc:1035] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.167573 9478 raft_consensus.cc:2932] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:09.171769 2071 raft_consensus.cc:2932] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:09.173203 4834 raft_consensus.cc:2932] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:09.175668 9766 raft_consensus.cc:2932] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.180598 1943 raft_consensus.cc:2932] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.180702 2204 raft_consensus.cc:2932] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.189953 1669 catalog_manager.cc:5484] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.190058 1663 catalog_manager.cc:5484] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
W20240503 08:13:09.211519 1878 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.196:42183, user_credentials={real_user=slave}} blocked reactor thread for 65195.2us
I20240503 08:13:09.218807 4965 raft_consensus.cc:2932] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.236078 1812 raft_consensus.cc:2932] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:09.239122 4699 consensus_queue.cc:237] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:09.241677 9650 raft_consensus.cc:1050] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5: attempting to promote NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a to VOTER
I20240503 08:13:09.242571 9650 consensus_queue.cc:237] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:13:09.243266 4833 raft_consensus.cc:1261] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.245287 2071 raft_consensus.cc:1261] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.245805 9680 consensus_queue.cc:1035] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.246843 9680 consensus_queue.cc:1035] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.250897 9841 raft_consensus.cc:2932] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:09.252568 2071 raft_consensus.cc:2932] T 8f3e931766d44f0aaf902cf0e950f83b P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:09.255409 9786 raft_consensus.cc:1050] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER 184c543cb05d49cc98ba1c56617be7a5 to VOTER
I20240503 08:13:09.256278 9786 consensus_queue.cc:237] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:09.256943 1943 raft_consensus.cc:1261] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240503 08:13:09.257340 4900 consensus_peers.cc:485] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.257681 9509 consensus_queue.cc:1035] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.265566 2073 raft_consensus.cc:1261] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 00b61625d8a7460892c118a2570afda5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.266374 9650 consensus_queue.cc:1035] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.272774 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f3e931766d44f0aaf902cf0e950f83b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:09.273994 4699 raft_consensus.cc:1261] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.274755 9786 consensus_queue.cc:1035] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 08:13:09.275393 1748 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.194:41975, user_credentials={real_user=slave}} blocked reactor thread for 87574.3us
I20240503 08:13:09.275525 1669 catalog_manager.cc:5484] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:09.279137 4965 raft_consensus.cc:1203] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:09.279695 2204 raft_consensus.cc:1261] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.281208 9786 consensus_queue.cc:1035] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.283721 4834 raft_consensus.cc:2932] T 8f3e931766d44f0aaf902cf0e950f83b P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:09.284668 9697 ts_tablet_manager.cc:902] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e: Initiating tablet copy from peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:13:09.287900 1812 raft_consensus.cc:1261] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.290261 9786 consensus_queue.cc:1035] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.297380 1812 consensus_queue.cc:237] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:09.301769 2201 raft_consensus.cc:1261] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.302599 9769 consensus_queue.cc:1035] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.307024 9769 raft_consensus.cc:2932] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.311781 2201 raft_consensus.cc:2932] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.322278 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3e931766d44f0aaf902cf0e950f83b with delete_type TABLET_DATA_TOMBSTONED (TS b8f581390da449ccb9e299011296405e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48256
I20240503 08:13:09.322434 1669 catalog_manager.cc:5484] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New cstate: current_term: 1 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.326591 1658 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5785af2b9ba34a3b95c5ffdb024b1c8f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:09.326743 1807 raft_consensus.cc:1203] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:09.328401 4945 tablet_service.cc:1514] Processing DeleteTablet for tablet 5785af2b9ba34a3b95c5ffdb024b1c8f with delete_type TABLET_DATA_TOMBSTONED (TS 00b61625d8a7460892c118a2570afda5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47168
I20240503 08:13:09.329825 1943 raft_consensus.cc:1261] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 08:13:09.334765 2137 consensus_peers.cc:462] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.336063 9509 raft_consensus.cc:2932] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:09.340271 9853 consensus_queue.cc:1035] T 5785af2b9ba34a3b95c5ffdb024b1c8f P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.341176 9518 raft_consensus.cc:1050] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:13:09.342041 9518 consensus_queue.cc:237] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:09.343384 1943 raft_consensus.cc:2932] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00b61625d8a7460892c118a2570afda5 (127.27.157.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.346969 9870 tablet_replica.cc:330] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5: stopping tablet replica
I20240503 08:13:09.348194 9870 raft_consensus.cc:2229] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:09.349355 9868 tablet_replica.cc:330] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:13:09.349587 2071 raft_consensus.cc:2932] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:09.350356 9868 raft_consensus.cc:2229] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:09.352634 9870 raft_consensus.cc:2258] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.352876 9868 raft_consensus.cc:2258] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.354849 9870 ts_tablet_manager.cc:1875] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:09.356609 1942 raft_consensus.cc:2932] T 438146d5fa194156b929d7190b11a5de P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:09.357363 9786 raft_consensus.cc:2932] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.360520 1812 raft_consensus.cc:2932] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.365430 1663 catalog_manager.cc:5484] T 438146d5fa194156b929d7190b11a5de P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:09.368079 9868 ts_tablet_manager.cc:1875] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:09.372207 1665 catalog_manager.cc:5484] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.375195 4965 raft_consensus.cc:1203] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 08:13:09.386592 9870 ts_tablet_manager.cc:1888] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:09.386780 9870 log.cc:1196] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 00b61625d8a7460892c118a2570afda5: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/wals/5785af2b9ba34a3b95c5ffdb024b1c8f
I20240503 08:13:09.387689 1657 catalog_manager.cc:4830] TS 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283): tablet 5785af2b9ba34a3b95c5ffdb024b1c8f (table rebalancer_extra_table_42 [id=682625ce97fd4bbaa5012a004977db42]) successfully deleted
I20240503 08:13:09.388360 2204 raft_consensus.cc:2932] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.391850 4700 raft_consensus.cc:2932] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.419692 2071 consensus_queue.cc:237] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:09.422681 2204 raft_consensus.cc:1261] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.422724 1812 raft_consensus.cc:1261] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.423561 8881 consensus_queue.cc:1035] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.424314 8881 consensus_queue.cc:1035] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:09.424970 1877 consensus_peers.cc:462] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.428027 9786 raft_consensus.cc:2932] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.430127 9697 tablet_copy_client.cc:308] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.157.195:43117
I20240503 08:13:09.430711 2201 raft_consensus.cc:2932] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.431612 4699 raft_consensus.cc:1261] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.432271 9518 consensus_queue.cc:1035] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.434190 2071 raft_consensus.cc:1261] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.435253 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0c3070c262ed420a91ac0ed063d16471 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:09.435585 9241 consensus_queue.cc:1035] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.435791 1812 raft_consensus.cc:2932] T 0c3070c262ed420a91ac0ed063d16471 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.436836 4965 raft_consensus.cc:1261] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.439370 9518 consensus_queue.cc:1035] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.441679 9241 raft_consensus.cc:482] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.441921 9241 raft_consensus.cc:504] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:09.442831 9241 leader_election.cc:288] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:13:09.443267 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84831077defe434c9f29eecfae7b44c8" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:13:09.447985 1669 catalog_manager.cc:5484] T 0c3070c262ed420a91ac0ed063d16471 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } } }
I20240503 08:13:09.453827 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c3070c262ed420a91ac0ed063d16471 with delete_type TABLET_DATA_TOMBSTONED (TS 31f41542452b4b1eb994ff4223d039fc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57094
I20240503 08:13:09.457679 9241 raft_consensus.cc:1050] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 315f3f0bcbf447a4b93b1aafcfbf0f5a to VOTER
I20240503 08:13:09.458933 9241 consensus_queue.cc:237] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
W20240503 08:13:09.466485 4767 leader_election.cc:334] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.467288 4767 leader_election.cc:302] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f24648c92c384f1ba349eee801f985e2; no voters: 43e2e01cdd314f9c872c4336ef74250a, 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:09.467877 9518 raft_consensus.cc:2726] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:09.471827 9882 tablet_replica.cc:330] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:13:09.472004 9875 raft_consensus.cc:2932] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.472829 9882 raft_consensus.cc:2229] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:09.475244 9882 raft_consensus.cc:2258] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.477411 9882 ts_tablet_manager.cc:1875] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:09.477644 4965 raft_consensus.cc:2932] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.479951 9868 ts_tablet_manager.cc:1888] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:09.481662 2093 tablet_copy_service.cc:140] P 43e2e01cdd314f9c872c4336ef74250a: Received BeginTabletCopySession request for tablet c2d33b2dc884448483072fa6ab825f0b from peer b8f581390da449ccb9e299011296405e ({username='slave'} at 127.27.157.193:45249)
I20240503 08:13:09.481884 2093 tablet_copy_service.cc:161] P 43e2e01cdd314f9c872c4336ef74250a: Beginning new tablet copy session on tablet c2d33b2dc884448483072fa6ab825f0b from peer b8f581390da449ccb9e299011296405e at {username='slave'} at 127.27.157.193:45249: session id = b8f581390da449ccb9e299011296405e-c2d33b2dc884448483072fa6ab825f0b
I20240503 08:13:09.483510 2071 raft_consensus.cc:2932] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.484678 2093 tablet_copy_source_session.cc:205] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 08:13:09.491978 9868 log.cc:1196] T 8f3e931766d44f0aaf902cf0e950f83b P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/8f3e931766d44f0aaf902cf0e950f83b
I20240503 08:13:09.489017 9697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d33b2dc884448483072fa6ab825f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:09.494565 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 8f3e931766d44f0aaf902cf0e950f83b (table rebalancer_extra_table_48 [id=f0aa7c71965d48539c0ac7118319411e]) successfully deleted
I20240503 08:13:09.496258 1943 raft_consensus.cc:1261] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.497421 1663 catalog_manager.cc:5484] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.498445 9874 consensus_queue.cc:1035] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 08:13:09.501072 4767 consensus_peers.cc:485] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.504043 9882 ts_tablet_manager.cc:1888] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:09.508458 9697 tablet_copy_client.cc:783] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 0 data blocks...
I20240503 08:13:09.508785 9882 log.cc:1196] T 0c3070c262ed420a91ac0ed063d16471 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/0c3070c262ed420a91ac0ed063d16471
I20240503 08:13:09.509924 9697 tablet_copy_client.cc:647] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e: tablet copy: Starting download of 1 WAL segments...
I20240503 08:13:09.510646 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 0c3070c262ed420a91ac0ed063d16471 (table rebalancer_extra_table_46 [id=734b33fe6b21405a806dd0d9942ffd95]) successfully deleted
I20240503 08:13:09.510915 4700 raft_consensus.cc:1261] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.511658 9876 raft_consensus.cc:482] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.511852 9876 raft_consensus.cc:504] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:13:09.513183 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f567b4ad00e242ba94187bdaeda71d66" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:13:09.513947 9876 leader_election.cc:288] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:13:09.514134 9876 consensus_queue.cc:1035] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 08:13:09.514426 4767 leader_election.cc:334] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.514603 4767 leader_election.cc:302] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f24648c92c384f1ba349eee801f985e2; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:13:09.515458 4901 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.196:42183, user_credentials={real_user=slave}} blocked reactor thread for 81127.1us
I20240503 08:13:09.516003 9697 tablet_copy_client.cc:523] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:13:09.516067 9241 raft_consensus.cc:2726] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:09.519837 9697 tablet_bootstrap.cc:492] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e: Bootstrap starting.
I20240503 08:13:09.526321 7964 raft_consensus.cc:482] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.526546 7964 raft_consensus.cc:504] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:13:09.532348 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f567b4ad00e242ba94187bdaeda71d66" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:13:09.532740 4834 raft_consensus.cc:2454] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:13:09.534168 1880 leader_election.cc:302] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:13:09.534615 9617 raft_consensus.cc:2781] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:13:09.534726 9617 raft_consensus.cc:482] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.534838 9617 raft_consensus.cc:3037] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.538781 9617 raft_consensus.cc:504] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:13:09.539690 7964 leader_election.cc:288] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
W20240503 08:13:09.545585 1877 leader_election.cc:334] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.547673 9617 leader_election.cc:288] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:13:09.547174 4700 raft_consensus.cc:2932] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
W20240503 08:13:09.550318 1877 leader_election.cc:334] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.554874 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f567b4ad00e242ba94187bdaeda71d66" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:13:09.555151 4834 raft_consensus.cc:3037] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.559646 4834 raft_consensus.cc:2454] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 2.
I20240503 08:13:09.560256 1880 leader_election.cc:302] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, f24648c92c384f1ba349eee801f985e2; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:09.560927 9617 raft_consensus.cc:2781] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:09.561218 9617 raft_consensus.cc:686] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:13:09.561491 9617 consensus_queue.cc:237] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } }
I20240503 08:13:09.565047 9876 raft_consensus.cc:2932] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:09.570895 1665 catalog_manager.cc:5484] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 1 to 2, leader changed from 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 2 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:09.573614 4700 raft_consensus.cc:2932] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:09.581322 1943 raft_consensus.cc:2932] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:09.583205 9786 raft_consensus.cc:1050] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER f24648c92c384f1ba349eee801f985e2 to VOTER
I20240503 08:13:09.584201 9786 consensus_queue.cc:237] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:13:09.585441 1665 catalog_manager.cc:5484] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 2 to 3, 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:09.595352 4833 consensus_queue.cc:237] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:09.598956 4965 raft_consensus.cc:1261] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.599242 2073 raft_consensus.cc:1261] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.599900 9876 consensus_queue.cc:1035] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.600317 9518 consensus_queue.cc:1035] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.605960 9478 raft_consensus.cc:482] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.606209 9478 raft_consensus.cc:504] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:13:09.606663 9697 tablet_bootstrap.cc:492] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:13:09.607050 9697 tablet_bootstrap.cc:492] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e: Bootstrap complete.
I20240503 08:13:09.607134 9478 leader_election.cc:288] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:13:09.607362 9697 ts_tablet_manager.cc:1367] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e: Time spent bootstrapping tablet: real 0.088s user 0.047s sys 0.031s
I20240503 08:13:09.608075 9697 raft_consensus.cc:348] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:09.608310 9697 raft_consensus.cc:729] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f581390da449ccb9e299011296405e, State: Initialized, Role: LEARNER
I20240503 08:13:09.608500 9697 consensus_queue.cc:260] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:09.609174 9697 ts_tablet_manager.cc:1398] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240503 08:13:09.609462 4635 leader_election.cc:334] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.609867 2093 tablet_copy_service.cc:342] P 43e2e01cdd314f9c872c4336ef74250a: Request end of tablet copy session b8f581390da449ccb9e299011296405e-c2d33b2dc884448483072fa6ab825f0b received from {username='slave'} at 127.27.157.193:45249
I20240503 08:13:09.609987 2093 tablet_copy_service.cc:433] P 43e2e01cdd314f9c872c4336ef74250a: ending tablet copy session b8f581390da449ccb9e299011296405e-c2d33b2dc884448483072fa6ab825f0b on tablet c2d33b2dc884448483072fa6ab825f0b with peer b8f581390da449ccb9e299011296405e
I20240503 08:13:09.610559 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa23eb4acce4d6f8733eb80689c57e6" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:13:09.611440 4637 leader_election.cc:302] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:09.611953 9680 raft_consensus.cc:2726] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:09.613226 9518 raft_consensus.cc:2932] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.613809 9478 raft_consensus.cc:1050] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc: attempting to promote NON_VOTER 43e2e01cdd314f9c872c4336ef74250a to VOTER
I20240503 08:13:09.614672 9478 consensus_queue.cc:237] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:09.616187 2073 raft_consensus.cc:2932] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.617233 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 826e7d6e6d2947fbb2a178e8e2a32d56 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:09.618538 4965 raft_consensus.cc:2932] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.633529 4833 consensus_queue.cc:237] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:13:09.636301 1669 catalog_manager.cc:5484] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.637086 1943 raft_consensus.cc:1261] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.637818 9518 consensus_queue.cc:1035] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.640297 4699 raft_consensus.cc:1261] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.641052 9733 consensus_queue.cc:1035] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.641996 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet 826e7d6e6d2947fbb2a178e8e2a32d56 with delete_type TABLET_DATA_TOMBSTONED (TS 31f41542452b4b1eb994ff4223d039fc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57094
I20240503 08:13:09.642386 9882 tablet_replica.cc:330] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:13:09.642617 9882 raft_consensus.cc:2229] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:09.642916 9882 raft_consensus.cc:2258] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:09.643853 9882 ts_tablet_manager.cc:1875] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:09.644351 9241 raft_consensus.cc:2932] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:09.646100 4699 raft_consensus.cc:2932] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:09.648864 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 51f3edb7a72c4ba083429c150fa73363 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:09.653048 9882 ts_tablet_manager.cc:1888] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:09.655117 1942 raft_consensus.cc:2932] T 51f3edb7a72c4ba083429c150fa73363 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:09.660414 1665 catalog_manager.cc:5484] T 51f3edb7a72c4ba083429c150fa73363 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } } }
I20240503 08:13:09.661868 9643 raft_consensus.cc:482] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.662114 9643 raft_consensus.cc:504] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:13:09.662993 9643 leader_election.cc:288] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:13:09.663817 4833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae7520f758c47a39cfa3aaf59ebc674" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
W20240503 08:13:09.665527 2007 leader_election.cc:334] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.665701 2007 leader_election.cc:302] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a; no voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2
I20240503 08:13:09.666205 9643 raft_consensus.cc:2726] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:13:09.669159 1656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.670521 1812 raft_consensus.cc:1203] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 08:13:09.678457 1656 catalog_manager.cc:4573] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 51f3edb7a72c4ba083429c150fa73363: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.679348 9882 log.cc:1196] T 826e7d6e6d2947fbb2a178e8e2a32d56 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/826e7d6e6d2947fbb2a178e8e2a32d56
I20240503 08:13:09.681663 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet 826e7d6e6d2947fbb2a178e8e2a32d56 (table rebalancer_extra_table_34 [id=90ccd16da234473f86755bf3c2e4f43f]) successfully deleted
I20240503 08:13:09.683488 1812 raft_consensus.cc:1261] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.683851 9877 raft_consensus.cc:482] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.684111 9786 consensus_queue.cc:1035] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.684079 9877 raft_consensus.cc:504] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:13:09.685077 9877 leader_election.cc:288] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:09.687068 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ae3cc0ce534128b49f2da744d23144" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
W20240503 08:13:09.690758 4900 leader_election.cc:334] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.690980 4900 leader_election.cc:302] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 5f95b05d111e4090bbc59dec394ed0ab, b8f581390da449ccb9e299011296405e
I20240503 08:13:09.691605 9877 raft_consensus.cc:2726] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:09.692785 4699 raft_consensus.cc:1261] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.693773 9786 consensus_queue.cc:1035] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.694535 9617 raft_consensus.cc:482] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.694691 9617 raft_consensus.cc:504] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:13:09.696024 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa23eb4acce4d6f8733eb80689c57e6" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:13:09.696292 4700 raft_consensus.cc:2454] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:13:09.696765 1880 leader_election.cc:302] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:13:09.697177 7964 raft_consensus.cc:2781] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:13:09.697263 7964 raft_consensus.cc:482] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.697363 7964 raft_consensus.cc:3037] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.700249 7964 raft_consensus.cc:504] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
I20240503 08:13:09.701479 4833 raft_consensus.cc:1261] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2 [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.701936 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa23eb4acce4d6f8733eb80689c57e6" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:13:09.702239 4700 raft_consensus.cc:3037] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.703222 9786 consensus_queue.cc:1035] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.705619 4700 raft_consensus.cc:2454] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 2.
I20240503 08:13:09.706137 1880 leader_election.cc:302] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 31f41542452b4b1eb994ff4223d039fc; no voters:
I20240503 08:13:09.706476 9851 raft_consensus.cc:2781] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Leader election won for term 2
W20240503 08:13:09.710008 1877 leader_election.cc:334] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.710500 7964 leader_election.cc:288] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:09.715729 9895 raft_consensus.cc:482] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.715930 9895 raft_consensus.cc:504] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:09.716763 9895 leader_election.cc:288] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:13:09.719239 9680 raft_consensus.cc:1050] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
W20240503 08:13:09.719810 1877 leader_election.cc:334] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.720702 9617 leader_election.cc:288] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:09.720546 9680 consensus_queue.cc:237] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:09.724562 9769 raft_consensus.cc:482] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.724781 9769 raft_consensus.cc:504] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:13:09.725536 9769 leader_election.cc:288] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:09.726603 2073 raft_consensus.cc:1261] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEARNER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.727121 4965 raft_consensus.cc:1261] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240503 08:13:09.727829 1746 leader_election.cc:334] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.727843 9841 consensus_queue.cc:1035] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.728224 4965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ae3cc0ce534128b49f2da744d23144" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
I20240503 08:13:09.728446 4965 raft_consensus.cc:2454] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 1.
I20240503 08:13:09.728888 1747 leader_election.cc:302] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, b8f581390da449ccb9e299011296405e; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:09.729274 9769 raft_consensus.cc:2781] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:13:09.729353 9769 raft_consensus.cc:482] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.729439 9769 raft_consensus.cc:3037] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.731113 9852 raft_consensus.cc:482] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.731302 9852 raft_consensus.cc:504] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:13:09.732192 9852 leader_election.cc:288] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
I20240503 08:13:09.732270 9769 raft_consensus.cc:504] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:13:09.733952 9769 leader_election.cc:288] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 2 election: Requested vote from peers 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:09.734923 9841 consensus_queue.cc:1035] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 08:13:09.736585 4700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e697bc477a4899b69e4fdcc4e235e7" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:13:09.738262 4967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ae3cc0ce534128b49f2da744d23144" candidate_uuid: "b8f581390da449ccb9e299011296405e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5"
I20240503 08:13:09.738548 4967 raft_consensus.cc:3037] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.740500 4964 raft_consensus.cc:1261] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.741225 9841 consensus_queue.cc:1035] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.741819 4967 raft_consensus.cc:2454] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f581390da449ccb9e299011296405e in term 2.
W20240503 08:13:09.742441 4635 consensus_peers.cc:485] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:09.744096 2073 raft_consensus.cc:1261] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.744159 4834 raft_consensus.cc:1261] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:09.744634 9680 consensus_queue.cc:1035] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.745227 9841 consensus_queue.cc:1035] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.751495 1747 leader_election.cc:302] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, b8f581390da449ccb9e299011296405e; no voters:
W20240503 08:13:09.754467 1746 leader_election.cc:334] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.754621 1746 leader_election.cc:302] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f581390da449ccb9e299011296405e; no voters: 31f41542452b4b1eb994ff4223d039fc, 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:13:09.755005 1746 leader_election.cc:334] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.755401 9852 raft_consensus.cc:2781] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:09.755537 9852 raft_consensus.cc:686] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [term 2 LEADER]: Becoming Leader. State: Replica: b8f581390da449ccb9e299011296405e, State: Running, Role: LEADER
I20240503 08:13:09.755654 9853 raft_consensus.cc:2726] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:09.755777 9852 consensus_queue.cc:237] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:13:09.761687 7964 raft_consensus.cc:482] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.761919 7964 raft_consensus.cc:504] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:09.762756 7964 leader_election.cc:288] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:09.763312 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e0415e028b4303bcf4ab64f4ec6b54" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:13:09.765342 1663 catalog_manager.cc:5484] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e reported cstate change: term changed from 1 to 2, leader changed from 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) to b8f581390da449ccb9e299011296405e (127.27.157.193). New cstate: current_term: 2 leader_uuid: "b8f581390da449ccb9e299011296405e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:09.768760 9875 raft_consensus.cc:482] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.769078 9875 raft_consensus.cc:504] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
W20240503 08:13:09.769403 1877 leader_election.cc:334] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.769567 1877 leader_election.cc:302] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters: 43e2e01cdd314f9c872c4336ef74250a, 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:09.769902 9875 leader_election.cc:288] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
W20240503 08:13:09.771744 4767 leader_election.cc:334] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.772184 7964 raft_consensus.cc:2726] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:09.773224 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae7520f758c47a39cfa3aaf59ebc674" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:13:09.773582 2074 raft_consensus.cc:2454] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:13:09.775063 4768 leader_election.cc:302] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, f24648c92c384f1ba349eee801f985e2; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:09.775648 9875 raft_consensus.cc:2781] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:13:09.775816 9875 raft_consensus.cc:482] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.776062 9875 raft_consensus.cc:3037] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.779985 9875 raft_consensus.cc:504] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
W20240503 08:13:09.782501 4767 leader_election.cc:334] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.783452 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae7520f758c47a39cfa3aaf59ebc674" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:13:09.783748 2074 raft_consensus.cc:3037] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.789130 2074 raft_consensus.cc:2454] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 2.
I20240503 08:13:09.791376 4768 leader_election.cc:302] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, f24648c92c384f1ba349eee801f985e2; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:09.791801 9876 raft_consensus.cc:2781] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:09.791965 9876 raft_consensus.cc:686] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [term 2 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:13:09.792281 9876 consensus_queue.cc:237] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:13:09.795018 9875 leader_election.cc:288] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:13:09.810226 9478 raft_consensus.cc:2932] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
W20240503 08:13:09.813997 2137 leader_election.cc:334] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.814488 9851 raft_consensus.cc:686] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:13:09.814828 9851 consensus_queue.cc:237] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } }
W20240503 08:13:09.815263 2137 consensus_peers.cc:462] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.818542 9844 raft_consensus.cc:2932] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.820490 4831 raft_consensus.cc:2932] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.826308 4967 raft_consensus.cc:2932] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.836735 1663 catalog_manager.cc:5484] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.839401 4965 raft_consensus.cc:2932] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:09.844816 9643 raft_consensus.cc:2932] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:09.845677 8881 raft_consensus.cc:482] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.845382 2074 raft_consensus.cc:2932] T 7a792a309b3c43f482b4adb0fbcfadde P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:09.845849 8881 raft_consensus.cc:504] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:09.846664 8881 leader_election.cc:288] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:09.848704 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e0415e028b4303bcf4ab64f4ec6b54" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
I20240503 08:13:09.848970 1942 raft_consensus.cc:2454] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:13:09.850337 2009 leader_election.cc:302] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:13:09.850016 4699 raft_consensus.cc:2932] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:09.851094 8881 raft_consensus.cc:2781] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:13:09.851208 8881 raft_consensus.cc:482] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.851311 8881 raft_consensus.cc:3037] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:13:09.852715 2007 leader_election.cc:334] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.854244 8881 raft_consensus.cc:504] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:09.855752 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e0415e028b4303bcf4ab64f4ec6b54" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a"
I20240503 08:13:09.855969 1942 raft_consensus.cc:3037] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.858830 1942 raft_consensus.cc:2454] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 2.
I20240503 08:13:09.859915 2009 leader_election.cc:302] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a; no voters:
I20240503 08:13:09.860493 1669 catalog_manager.cc:5484] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 1 to 2, leader changed from 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 2 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:09.870528 8881 leader_election.cc:288] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 election: Requested vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:09.870795 8881 raft_consensus.cc:2781] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:09.870714 4700 consensus_queue.cc:237] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:09.871106 8881 raft_consensus.cc:686] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 2 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:13:09.871491 8881 consensus_queue.cc:237] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } }
I20240503 08:13:09.876542 9518 raft_consensus.cc:1050] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:13:09.876641 2071 raft_consensus.cc:1261] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.877364 9841 consensus_queue.cc:1035] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:09.877526 9518 consensus_queue.cc:237] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:09.877614 1665 catalog_manager.cc:5484] T 7a792a309b3c43f482b4adb0fbcfadde P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 2 to 3, 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } } }
I20240503 08:13:09.878835 2071 raft_consensus.cc:2932] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
W20240503 08:13:09.882082 2007 leader_election.cc:334] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.888054 4965 raft_consensus.cc:1261] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:09.888932 9841 consensus_queue.cc:1035] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:09.892194 1663 catalog_manager.cc:5484] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc reported cstate change: config changed from index 2 to 3, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:09.893039 9841 raft_consensus.cc:2932] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.896414 4967 raft_consensus.cc:2932] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.897735 9786 raft_consensus.cc:482] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.897990 9786 raft_consensus.cc:504] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:09.898975 9786 leader_election.cc:288] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
I20240503 08:13:09.901278 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48714749469474886f9e1d7e7635915" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" is_pre_election: true
W20240503 08:13:09.902709 2007 leader_election.cc:334] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.906265 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e613a6cbac2c4ad9b71442baa71ba9a3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:09.909953 1665 catalog_manager.cc:5484] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New cstate: current_term: 1 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.916851 9844 raft_consensus.cc:482] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.917069 9844 raft_consensus.cc:504] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:13:09.917919 9844 leader_election.cc:288] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:13:09.919871 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e697bc477a4899b69e4fdcc4e235e7" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e" is_pre_election: true
I20240503 08:13:09.920156 1809 raft_consensus.cc:2454] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 1.
W20240503 08:13:09.920727 4635 leader_election.cc:334] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.921069 4637 leader_election.cc:302] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:09.921413 9844 raft_consensus.cc:2781] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:13:09.921581 9844 raft_consensus.cc:482] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.921711 9844 raft_consensus.cc:3037] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.926726 9874 raft_consensus.cc:482] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.926462 2071 raft_consensus.cc:2932] T e613a6cbac2c4ad9b71442baa71ba9a3 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f24648c92c384f1ba349eee801f985e2 (127.27.157.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:09.926939 9874 raft_consensus.cc:504] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:09.927886 9874 leader_election.cc:288] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
W20240503 08:13:09.929984 1877 consensus_peers.cc:462] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:09.932827 2140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.198:46559, user_credentials={real_user=slave}} blocked reactor thread for 104322us
I20240503 08:13:09.939213 2139 leader_election.cc:302] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:13:09.939822 4767 leader_election.cc:334] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.940385 9890 raft_consensus.cc:2726] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:09.940867 9733 raft_consensus.cc:1050] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2: attempting to promote NON_VOTER 00b61625d8a7460892c118a2570afda5 to VOTER
I20240503 08:13:09.941797 9733 consensus_queue.cc:237] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:09.944876 9844 raft_consensus.cc:504] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:13:09.946887 9844 leader_election.cc:288] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
W20240503 08:13:09.947938 4635 leader_election.cc:334] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.949836 2074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cda734f360c4d3eac88a6517028e49d" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a" is_pre_election: true
I20240503 08:13:09.950136 2074 raft_consensus.cc:2454] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:13:09.950616 4768 leader_election.cc:302] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, f24648c92c384f1ba349eee801f985e2; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:09.951465 9876 raft_consensus.cc:2781] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:13:09.951498 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4e697bc477a4899b69e4fdcc4e235e7" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b8f581390da449ccb9e299011296405e"
I20240503 08:13:09.951601 9876 raft_consensus.cc:482] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.951704 9876 raft_consensus.cc:3037] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.951715 1809 raft_consensus.cc:3037] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.953452 9786 raft_consensus.cc:482] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:09.953274 1669 catalog_manager.cc:5484] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 1 to 2, leader changed from 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 2 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:09.953620 9786 raft_consensus.cc:504] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:09.954281 9786 leader_election.cc:288] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:09.954406 1809 raft_consensus.cc:2454] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31f41542452b4b1eb994ff4223d039fc in term 2.
W20240503 08:13:09.957114 2007 leader_election.cc:334] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.957494 4637 leader_election.cc:302] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, b8f581390da449ccb9e299011296405e; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:09.958060 9876 raft_consensus.cc:504] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:09.958463 9478 raft_consensus.cc:2781] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:09.958699 9478 raft_consensus.cc:686] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [term 2 LEADER]: Becoming Leader. State: Replica: 31f41542452b4b1eb994ff4223d039fc, State: Running, Role: LEADER
I20240503 08:13:09.958994 9478 consensus_queue.cc:237] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } }
I20240503 08:13:09.959259 9876 leader_election.cc:288] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
W20240503 08:13:09.962280 4767 leader_election.cc:334] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:09.968955 2073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cda734f360c4d3eac88a6517028e49d" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "43e2e01cdd314f9c872c4336ef74250a"
I20240503 08:13:09.969451 2073 raft_consensus.cc:3037] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:09.977106 2071 consensus_queue.cc:237] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:13:09.993304 4813 tablet_service.cc:1514] Processing DeleteTablet for tablet e613a6cbac2c4ad9b71442baa71ba9a3 with delete_type TABLET_DATA_TOMBSTONED (TS f24648c92c384f1ba349eee801f985e2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33952
I20240503 08:13:10.007057 9919 raft_consensus.cc:482] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:10.007268 9919 raft_consensus.cc:504] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
W20240503 08:13:10.009821 1877 leader_election.cc:334] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:10.010182 9919 leader_election.cc:288] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:13:10.035120 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48714749469474886f9e1d7e7635915" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:13:10.035475 2204 raft_consensus.cc:2454] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 1.
I20240503 08:13:10.036406 1878 leader_election.cc:302] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:10.036671 4831 raft_consensus.cc:1261] T f567b4ad00e242ba94187bdaeda71d66 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:13:10.036851 9921 raft_consensus.cc:2781] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:13:10.036936 9921 raft_consensus.cc:482] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:10.037012 9921 raft_consensus.cc:3037] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:10.040041 9921 raft_consensus.cc:504] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:10.040760 9921 leader_election.cc:288] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 election: Requested vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:13:10.040925 9921 consensus_queue.cc:1035] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 08:13:10.042444 1877 consensus_peers.cc:485] T f567b4ad00e242ba94187bdaeda71d66 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:10.043500 1877 leader_election.cc:334] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:10.044236 2204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48714749469474886f9e1d7e7635915" candidate_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:13:10.044482 2204 raft_consensus.cc:3037] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:10.047384 2204 raft_consensus.cc:2454] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315f3f0bcbf447a4b93b1aafcfbf0f5a in term 2.
I20240503 08:13:10.047914 1878 leader_election.cc:302] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:10.048193 9921 raft_consensus.cc:2781] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:10.048322 9921 raft_consensus.cc:686] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 LEADER]: Becoming Leader. State: Replica: 315f3f0bcbf447a4b93b1aafcfbf0f5a, State: Running, Role: LEADER
I20240503 08:13:10.048584 9921 consensus_queue.cc:237] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:10.051071 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cda734f360c4d3eac88a6517028e49d" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:13:10.051447 9927 tablet_replica.cc:330] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2: stopping tablet replica
I20240503 08:13:10.051539 4834 raft_consensus.cc:2379] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43e2e01cdd314f9c872c4336ef74250a in current term 2: Already voted for candidate f24648c92c384f1ba349eee801f985e2 in this term.
I20240503 08:13:10.051688 9927 raft_consensus.cc:2229] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:10.051827 4698 raft_consensus.cc:1261] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:10.052006 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84831077defe434c9f29eecfae7b44c8" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
I20240503 08:13:10.052188 9927 raft_consensus.cc:2258] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:10.052261 4835 raft_consensus.cc:2454] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 1.
I20240503 08:13:10.053083 4830 raft_consensus.cc:1261] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:10.054333 9927 ts_tablet_manager.cc:1875] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:10.058439 1809 raft_consensus.cc:1261] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:10.059473 9518 consensus_queue.cc:1035] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:10.061612 9915 consensus_queue.cc:1035] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:10.064955 2010 leader_election.cc:302] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a; no voters: 5f95b05d111e4090bbc59dec394ed0ab, f24648c92c384f1ba349eee801f985e2
I20240503 08:13:10.066073 4967 raft_consensus.cc:1261] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5 [term 1 LEARNER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:10.066150 2071 raft_consensus.cc:1261] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:10.066490 2010 leader_election.cc:302] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, f24648c92c384f1ba349eee801f985e2; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:10.066824 9917 raft_consensus.cc:2781] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:13:10.066965 9917 raft_consensus.cc:482] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f95b05d111e4090bbc59dec394ed0ab)
I20240503 08:13:10.067041 9914 consensus_queue.cc:1035] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:10.067077 9917 raft_consensus.cc:3037] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:10.069404 9876 consensus_queue.cc:1035] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:10.069444 9874 consensus_queue.cc:1035] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:10.071828 4833 raft_consensus.cc:2932] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f24648c92c384f1ba349eee801f985e2 (127.27.157.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:10.085747 9917 raft_consensus.cc:504] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:10.095420 9917 leader_election.cc:288] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 election: Requested vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
W20240503 08:13:10.099546 2007 leader_election.cc:334] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:10.099524 9916 raft_consensus.cc:2932] T 12b0c4a0b0564a279806d3fd6f981eed P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:10.100723 4830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84831077defe434c9f29eecfae7b44c8" candidate_uuid: "43e2e01cdd314f9c872c4336ef74250a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2"
I20240503 08:13:10.101110 4830 raft_consensus.cc:3037] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:10.104707 4834 raft_consensus.cc:2932] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:10.108366 1812 raft_consensus.cc:1261] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:10.109232 4830 raft_consensus.cc:2454] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43e2e01cdd314f9c872c4336ef74250a in term 2.
I20240503 08:13:10.109488 9518 consensus_queue.cc:1035] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:10.114202 2010 leader_election.cc:302] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, f24648c92c384f1ba349eee801f985e2; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:10.114647 9917 raft_consensus.cc:2781] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:10.114881 9917 raft_consensus.cc:686] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [term 2 LEADER]: Becoming Leader. State: Replica: 43e2e01cdd314f9c872c4336ef74250a, State: Running, Role: LEADER
I20240503 08:13:10.115360 9917 consensus_queue.cc:237] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:10.126773 2204 raft_consensus.cc:1261] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:10.129091 2073 raft_consensus.cc:2454] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 2.
I20240503 08:13:10.132838 9733 consensus_queue.cc:1035] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:10.140705 4700 raft_consensus.cc:1261] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:10.141629 4698 raft_consensus.cc:2932] T 12b0c4a0b0564a279806d3fd6f981eed P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:10.142484 9876 consensus_queue.cc:1035] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:10.147584 9876 raft_consensus.cc:2932] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:10.148463 4700 raft_consensus.cc:2932] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:10.153121 4965 raft_consensus.cc:2932] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
W20240503 08:13:10.164323 1746 consensus_peers.cc:485] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:10.168696 1667 catalog_manager.cc:5484] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:10.176245 4967 raft_consensus.cc:1261] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Refusing update from remote peer b8f581390da449ccb9e299011296405e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:13:10.178464 9853 consensus_queue.cc:1035] T 09ae3cc0ce534128b49f2da744d23144 P b8f581390da449ccb9e299011296405e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:10.195972 4700 raft_consensus.cc:1261] T 1aa23eb4acce4d6f8733eb80689c57e6 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:13:10.197106 9921 consensus_queue.cc:1035] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:10.206347 9643 raft_consensus.cc:1050] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:13:10.227211 9643 consensus_queue.cc:237] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:10.242127 9915 raft_consensus.cc:2726] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:10.173483 1665 catalog_manager.cc:5484] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc reported cstate change: term changed from 1 to 2, leader changed from 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) to 31f41542452b4b1eb994ff4223d039fc (127.27.157.198). New cstate: current_term: 2 leader_uuid: "31f41542452b4b1eb994ff4223d039fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:10.245076 4768 leader_election.cc:302] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43e2e01cdd314f9c872c4336ef74250a, f24648c92c384f1ba349eee801f985e2; no voters: 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:10.245889 9733 raft_consensus.cc:2781] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:10.246279 9733 raft_consensus.cc:686] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [term 2 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:13:10.248332 9733 consensus_queue.cc:237] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:10.249560 1809 raft_consensus.cc:3037] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Advancing to term 2
I20240503 08:13:10.251072 1812 raft_consensus.cc:2932] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 00b61625d8a7460892c118a2570afda5 (127.27.157.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
W20240503 08:13:10.251883 2137 consensus_peers.cc:462] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:10.256230 1877 consensus_peers.cc:485] T 1aa23eb4acce4d6f8733eb80689c57e6 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:10.266955 9874 raft_consensus.cc:2932] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.253614 1809 raft_consensus.cc:1261] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e [term 2 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:10.272862 1811 raft_consensus.cc:2932] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.283797 2204 raft_consensus.cc:2932] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.289808 1669 catalog_manager.cc:5484] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 1 to 2, leader changed from 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 2 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:10.290076 9915 consensus_queue.cc:1035] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: NON_VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20240503 08:13:10.290679 1670 catalog_manager.cc:5484] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.138134 1663 catalog_manager.cc:5484] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: term changed from 1 to 2, leader changed from 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) to 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194). New cstate: current_term: 2 leader_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:10.296605 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12b0c4a0b0564a279806d3fd6f981eed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 08:13:10.297678 2007 consensus_peers.cc:485] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:10.297315 2071 raft_consensus.cc:2932] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.298178 4698 raft_consensus.cc:1261] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:10.298825 9643 consensus_queue.cc:1035] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:10.304380 1807 raft_consensus.cc:1261] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e [term 1 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 08:13:10.305357 9916 consensus_queue.cc:1035] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:10.318825 9927 ts_tablet_manager.cc:1888] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:10.324596 9643 raft_consensus.cc:1050] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
W20240503 08:13:10.324796 9643 raft_consensus.cc:1054] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a: Unable to promote non-voter b8f581390da449ccb9e299011296405e: Illegal state: Leader has not yet committed an operation in its own term
I20240503 08:13:10.325049 9927 log.cc:1196] T e613a6cbac2c4ad9b71442baa71ba9a3 P f24648c92c384f1ba349eee801f985e2: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/wals/e613a6cbac2c4ad9b71442baa71ba9a3
I20240503 08:13:10.327966 1667 catalog_manager.cc:5484] T 12b0c4a0b0564a279806d3fd6f981eed P f24648c92c384f1ba349eee801f985e2 reported cstate change: config changed from index 3 to 4, VOTER b8f581390da449ccb9e299011296405e (127.27.157.193) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } }
I20240503 08:13:10.329319 1667 catalog_manager.cc:5484] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 1 to 2, leader changed from 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 2 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 08:13:10.341334 1877 consensus_peers.cc:462] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:10.341856 1793 tablet_service.cc:1514] Processing DeleteTablet for tablet 12b0c4a0b0564a279806d3fd6f981eed with delete_type TABLET_DATA_TOMBSTONED (TS b8f581390da449ccb9e299011296405e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48256
I20240503 08:13:10.341228 1668 catalog_manager.cc:5484] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: term changed from 1 to 2, leader changed from 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) to 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195). New cstate: current_term: 2 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:10.366920 4830 consensus_queue.cc:237] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:10.371033 1659 catalog_manager.cc:4830] TS f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): tablet e613a6cbac2c4ad9b71442baa71ba9a3 (table rebalancer_extra_table_44 [id=88bd25fd205149ce87c2edef8849bd38]) successfully deleted
I20240503 08:13:10.371425 4835 consensus_queue.cc:237] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:10.374059 1811 raft_consensus.cc:1261] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:10.374307 2202 raft_consensus.cc:1261] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:10.375156 4967 raft_consensus.cc:1261] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:10.375162 9733 consensus_queue.cc:1035] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:10.375594 8881 raft_consensus.cc:2932] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.377707 9518 consensus_queue.cc:1035] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:10.378496 9874 consensus_queue.cc:1035] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:10.379310 1811 raft_consensus.cc:2932] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.381192 1812 raft_consensus.cc:1261] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:10.381018 9875 raft_consensus.cc:2932] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.381695 9241 consensus_queue.cc:1035] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:10.381819 2202 raft_consensus.cc:2932] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.385566 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a03edb25e08e495c8dcb46adc9d8d2a4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:10.387025 1811 raft_consensus.cc:2932] T a03edb25e08e495c8dcb46adc9d8d2a4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.392896 1663 catalog_manager.cc:5484] T a03edb25e08e495c8dcb46adc9d8d2a4 P 184c543cb05d49cc98ba1c56617be7a5 reported cstate change: config changed from index 3 to 4, VOTER 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.395992 1668 catalog_manager.cc:5484] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a reported cstate change: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 08:13:10.398059 2007 consensus_peers.cc:485] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:10.404518 9518 raft_consensus.cc:2932] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:10.403901 2071 raft_consensus.cc:1261] T 8ae7520f758c47a39cfa3aaf59ebc674 P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:13:10.405428 4698 raft_consensus.cc:2932] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { replace: true } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.408107 9875 consensus_queue.cc:1035] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:10.412007 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet a03edb25e08e495c8dcb46adc9d8d2a4 with delete_type TABLET_DATA_TOMBSTONED (TS 43e2e01cdd314f9c872c4336ef74250a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35210
I20240503 08:13:10.413532 2073 consensus_queue.cc:237] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:10.415753 1659 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6b22a007a2540ea9364579ac680fcf4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 08:13:10.415966 4767 consensus_peers.cc:485] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:10.417441 1811 raft_consensus.cc:1261] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:10.418555 9915 consensus_queue.cc:1035] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:10.421203 4967 raft_consensus.cc:2932] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:10.430032 1668 catalog_manager.cc:5484] T d6b22a007a2540ea9364579ac680fcf4 P 00b61625d8a7460892c118a2570afda5 reported cstate change: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New cstate: current_term: 1 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:10.432286 1809 raft_consensus.cc:2932] T d6b22a007a2540ea9364579ac680fcf4 P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 31f41542452b4b1eb994ff4223d039fc (127.27.157.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } }
I20240503 08:13:10.441663 9915 raft_consensus.cc:2932] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.447702 1657 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2d33b2dc884448483072fa6ab825f0b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 08:13:10.448799 1811 raft_consensus.cc:2932] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.456564 1663 catalog_manager.cc:5484] T c2d33b2dc884448483072fa6ab825f0b P b8f581390da449ccb9e299011296405e reported cstate change: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New cstate: current_term: 1 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.460165 9958 tablet_replica.cc:330] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a: stopping tablet replica
I20240503 08:13:10.460662 9958 raft_consensus.cc:2229] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:10.461040 9958 raft_consensus.cc:2258] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 08:13:10.471781 4637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.193:41181, user_credentials={real_user=slave}} blocked reactor thread for 61947.9us
I20240503 08:13:10.474305 4683 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b22a007a2540ea9364579ac680fcf4 with delete_type TABLET_DATA_TOMBSTONED (TS 31f41542452b4b1eb994ff4223d039fc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57094
W20240503 08:13:10.481192 4635 consensus_peers.cc:485] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:10.482067 1807 raft_consensus.cc:1261] T c4e697bc477a4899b69e4fdcc4e235e7 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Refusing update from remote peer 31f41542452b4b1eb994ff4223d039fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:13:10.484054 4703 raft_consensus.cc:1261] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 08:13:10.484249 9955 tablet_replica.cc:330] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e: stopping tablet replica
I20240503 08:13:10.484541 9955 raft_consensus.cc:2229] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:10.484732 9955 pending_rounds.cc:62] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e: Trying to abort 1 pending ops.
I20240503 08:13:10.484835 9955 pending_rounds.cc:68] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023509461334429696 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "12b0c4a0b0564a279806d3fd6f981eed" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: NON_VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { replace: true } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } } }
I20240503 08:13:10.485226 9955 raft_consensus.cc:2894] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 08:13:10.485353 9955 raft_consensus.cc:2866] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 08:13:10.485503 9955 raft_consensus.cc:2258] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:10.485401 9844 consensus_queue.cc:1035] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:10.485886 9915 consensus_queue.cc:1035] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:10.487948 4703 raft_consensus.cc:2932] T c2d33b2dc884448483072fa6ab825f0b P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.490271 9958 ts_tablet_manager.cc:1875] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:10.492728 9963 tablet_replica.cc:330] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc: stopping tablet replica
I20240503 08:13:10.494180 1943 raft_consensus.cc:1261] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 08:13:10.495718 8881 consensus_queue.cc:1035] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 08:13:10.496011 9963 raft_consensus.cc:2229] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:13:10.496351 9963 raft_consensus.cc:2258] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:13:10.499431 9963 ts_tablet_manager.cc:1875] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:10.499578 9955 ts_tablet_manager.cc:1875] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:13:10.514990 9958 ts_tablet_manager.cc:1888] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
W20240503 08:13:10.524591 1877 consensus_peers.cc:485] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:10.526293 9958 log.cc:1196] T a03edb25e08e495c8dcb46adc9d8d2a4 P 43e2e01cdd314f9c872c4336ef74250a: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/wals/a03edb25e08e495c8dcb46adc9d8d2a4
I20240503 08:13:10.528708 1657 catalog_manager.cc:4830] TS 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): tablet a03edb25e08e495c8dcb46adc9d8d2a4 (table rebalancer_extra_table_44 [id=88bd25fd205149ce87c2edef8849bd38]) successfully deleted
I20240503 08:13:10.534287 9963 ts_tablet_manager.cc:1888] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:10.538853 9955 ts_tablet_manager.cc:1888] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 08:13:10.539095 9955 log.cc:1196] T 12b0c4a0b0564a279806d3fd6f981eed P b8f581390da449ccb9e299011296405e: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/wals/12b0c4a0b0564a279806d3fd6f981eed
I20240503 08:13:10.540479 4831 raft_consensus.cc:1261] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:13:10.542867 9963 log.cc:1196] T d6b22a007a2540ea9364579ac680fcf4 P 31f41542452b4b1eb994ff4223d039fc: Deleting WAL directory at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/wals/d6b22a007a2540ea9364579ac680fcf4
I20240503 08:13:10.545064 9917 consensus_queue.cc:1035] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:10.545298 1659 catalog_manager.cc:4830] TS 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): tablet d6b22a007a2540ea9364579ac680fcf4 (table rebalancer_extra_table_48 [id=f0aa7c71965d48539c0ac7118319411e]) successfully deleted
I20240503 08:13:10.546358 9917 raft_consensus.cc:1050] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a: attempting to promote NON_VOTER b8f581390da449ccb9e299011296405e to VOTER
I20240503 08:13:10.547163 9917 consensus_queue.cc:237] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } }
I20240503 08:13:10.554322 1812 raft_consensus.cc:1261] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e [term 2 LEARNER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:10.555699 1942 raft_consensus.cc:1261] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Refusing update from remote peer 43e2e01cdd314f9c872c4336ef74250a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:10.556344 9643 consensus_queue.cc:1035] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 08:13:10.559545 2007 consensus_peers.cc:485] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:10.560945 1658 catalog_manager.cc:4830] TS b8f581390da449ccb9e299011296405e (127.27.157.193:41181): tablet 12b0c4a0b0564a279806d3fd6f981eed (table rebalancer_extra_table_46 [id=734b33fe6b21405a806dd0d9942ffd95]) successfully deleted
I20240503 08:13:10.560897 9917 consensus_queue.cc:1035] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:10.565369 9915 raft_consensus.cc:2932] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.567813 1942 raft_consensus.cc:2932] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.573447 1811 raft_consensus.cc:2932] T a7e0415e028b4303bcf4ab64f4ec6b54 P b8f581390da449ccb9e299011296405e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
I20240503 08:13:10.575439 1667 catalog_manager.cc:5484] T a7e0415e028b4303bcf4ab64f4ec6b54 P 315f3f0bcbf447a4b93b1aafcfbf0f5a reported cstate change: config changed from index 2 to 4, b8f581390da449ccb9e299011296405e (127.27.157.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "43e2e01cdd314f9c872c4336ef74250a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } attrs { promote: false } } }
W20240503 08:13:10.597256 2007 consensus_peers.cc:485] T 84831077defe434c9f29eecfae7b44c8 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:10.604914 2202 raft_consensus.cc:1261] T b48714749469474886f9e1d7e7635915 P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Refusing update from remote peer 315f3f0bcbf447a4b93b1aafcfbf0f5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:13:10.606374 7964 consensus_queue.cc:1035] T b48714749469474886f9e1d7e7635915 P 315f3f0bcbf447a4b93b1aafcfbf0f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:13:10.643112 2137 consensus_peers.cc:462] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:10.698237 1656 catalog_manager.cc:4573] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c2d33b2dc884448483072fa6ab825f0b: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:10.757210 2071 raft_consensus.cc:1261] T 3cda734f360c4d3eac88a6517028e49d P 43e2e01cdd314f9c872c4336ef74250a [term 2 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 08:13:10.760215 9874 consensus_queue.cc:1035] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 08:13:10.769687 4767 consensus_peers.cc:485] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:10.798377 2140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.198:46559, user_credentials={real_user=slave}} blocked reactor thread for 56209.7us
W20240503 08:13:10.852622 1877 consensus_peers.cc:485] T 53197deeb8d7439c90b019a6a90857e0 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:10.925360 1877 consensus_peers.cc:462] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:10.970549 2007 consensus_peers.cc:485] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:10.979420 2007 consensus_peers.cc:485] T 89d9b1b3c23241d49bfcb914a5bd1777 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.106421 4900 consensus_peers.cc:485] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.133631 4767 consensus_peers.cc:485] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:11.148859 1924 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 08:13:11.152184 2137 consensus_peers.cc:462] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:11.161713 4683 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 08:13:11.174346 4945 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 08:13:11.220631 4900 consensus_peers.cc:485] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.234175 1877 consensus_peers.cc:485] T 674e558bf5db4f169f8cdcb08c40f5fb P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.234414 2007 consensus_peers.cc:485] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:11.252854 2054 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 08:13:11.262738 4900 consensus_peers.cc:485] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:11.274958 2183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 08:13:11.315701 4900 consensus_peers.cc:485] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:11.342676 1793 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 08:13:11.374102 4813 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 08:13:11.394076 4900 consensus_peers.cc:485] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.396943 1877 consensus_peers.cc:485] T fba1e1952da44c0e956be1ef7f592f08 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.398761 1877 consensus_peers.cc:485] T f4900ffdbc624376b32f054efdb100b5 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.399075 4767 consensus_peers.cc:485] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.407740 4900 consensus_peers.cc:485] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.427868 1877 consensus_peers.cc:462] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:11.441802 1746 consensus_peers.cc:485] T 3219159f92774adbb7ecaa77ac267021 P b8f581390da449ccb9e299011296405e -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.466378 1877 consensus_peers.cc:485] T 47dde43816964e89bfb235aad34e9583 P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.488052 1746 consensus_peers.cc:485] T 8fc8279b96f74e879b499b0143502c8f P b8f581390da449ccb9e299011296405e -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.498804 1746 consensus_peers.cc:485] T 9e371231b34f45dfac455bbbbe156263 P b8f581390da449ccb9e299011296405e -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.502400 4635 consensus_peers.cc:485] T 34e1c5dc09a54380a17f2cf08a473d57 P 31f41542452b4b1eb994ff4223d039fc -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.523087 4767 consensus_peers.cc:485] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.532847 2007 consensus_peers.cc:485] T 9cb8a87410b047548efdd973bbc86714 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.541078 4900 consensus_peers.cc:485] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.608101 4900 consensus_peers.cc:485] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.633075 4635 consensus_peers.cc:485] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.705353 2137 consensus_peers.cc:462] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:11.753659 2137 consensus_peers.cc:485] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.756879 4900 consensus_peers.cc:485] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskpimNqO/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 1
Maximum Replica Count | 43
Average Replica Count | 21.750000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.900000
stderr: W20240503 08:12:30.077569 5022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:12:30.078094 5022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:12:30.095526 5022 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
E20240503 08:12:30.696640 5140 ksck_remote.cc:634] unable to open table rebalance_test_table_2: Not found: the table does not exist: table_name: "rebalance_test_table_2"
I20240503 08:12:31.790524 5022 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 08:12:32.664800 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f1acf111b6bf49d39e0f38ca4a22f3bf: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:32.665163 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f62a9f6fd824e2e9bb6bbba182f6f56: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server 31f41542452b4b1eb994ff4223d039fc
W20240503 08:12:32.665319 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7f74723fa74a43f590ce741d046208a5: could not find any suitable replica to move from server b8f581390da449ccb9e299011296405e to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:32.665663 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f26b7d1f07364f2789326fba2a9c062a: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:32.665944 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ab330af65d474e5b9c89cd53a7f6a8b2: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:32.666105 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e422caa613e4bae95fb8412d4e942ca: could not find any suitable replica to move from server b8f581390da449ccb9e299011296405e to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:32.666369 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 81cf492c7a7940a7bf625ec8e2798f29: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server 31f41542452b4b1eb994ff4223d039fc
W20240503 08:12:32.666524 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 10478ffebe664fcda896887c04a61afa: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:32.666711 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f69a9daaaa3f44c28eac3e5c9d3b76d5: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:32.666926 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7f74723fa74a43f590ce741d046208a5: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 31f41542452b4b1eb994ff4223d039fc
W20240503 08:12:32.667105 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1d48efe8d72a4dfebb62a9559e3f4222: could not find any suitable replica to move from server 184c543cb05d49cc98ba1c56617be7a5 to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:32.667271 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9653225381b14c348f664a692ff48da1: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:32.667430 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b226fb18fc1f404ca9860f6b761a47b8: could not find any suitable replica to move from server b8f581390da449ccb9e299011296405e to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:32.667575 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 19626a36f6c54bd7b459b0de14aadbfc: could not find any suitable replica to move from server 184c543cb05d49cc98ba1c56617be7a5 to server 31f41542452b4b1eb994ff4223d039fc
W20240503 08:12:32.667714 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f26b7d1f07364f2789326fba2a9c062a: could not find any suitable replica to move from server 184c543cb05d49cc98ba1c56617be7a5 to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:32.667855 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37a3ac4b4467489788ff2543c2b3a179: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:32.668097 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1c93a14622e34eecb8d6bf508be45494: could not find any suitable replica to move from server b8f581390da449ccb9e299011296405e to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:32.668248 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c9883eb3c54f4eb89c079762678d5d4a: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:32.668383 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 10478ffebe664fcda896887c04a61afa: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:32.668573 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 81cf492c7a7940a7bf625ec8e2798f29: could not find any suitable replica to move from server 184c543cb05d49cc98ba1c56617be7a5 to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:32.668710 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f1acf111b6bf49d39e0f38ca4a22f3bf: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:32.668864 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f69a9daaaa3f44c28eac3e5c9d3b76d5: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 31f41542452b4b1eb994ff4223d039fc
I20240503 08:12:32.927992 5022 rebalancer_tool.cc:1074] tablet d3b2ebb6eee649199297eb13faa9e4cb: 'b8f581390da449ccb9e299011296405e' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:12:32.964288 5022 rebalancer_tool.cc:1074] tablet 48f411ec0bfc4a318b661441c49bd6ec: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:33.017712 5022 rebalancer_tool.cc:1074] tablet 868bcace806f4d32977b21c876c2362b: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:12:33.066972 5022 rebalancer_tool.cc:1074] tablet 1811bd652d2c4afb9f52bcd64ceb2acb: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:12:33.070859 5022 rebalancer_tool.cc:1084] tablet 595f7bc77944474db016cdb39965765b: 'b8f581390da449ccb9e299011296405e' -> 'f24648c92c384f1ba349eee801f985e2' move ignored: Not found: 595f7bc77944474db016cdb39965765b: tablet not found
I20240503 08:12:33.071044 5022 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:12:34.262622 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f62a9f6fd824e2e9bb6bbba182f6f56: could not find any suitable replica to move from server b8f581390da449ccb9e299011296405e to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:34.262967 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c9883eb3c54f4eb89c079762678d5d4a: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:34.271379 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f62a9f6fd824e2e9bb6bbba182f6f56: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:34.272044 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bc178b718fe049ae96bd1ff6f9794fba: could not find any suitable replica to move from server f24648c92c384f1ba349eee801f985e2 to server 184c543cb05d49cc98ba1c56617be7a5
W20240503 08:12:34.272919 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ab330af65d474e5b9c89cd53a7f6a8b2: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:34.273059 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e422caa613e4bae95fb8412d4e942ca: could not find any suitable replica to move from server b8f581390da449ccb9e299011296405e to server 31f41542452b4b1eb994ff4223d039fc
W20240503 08:12:34.273211 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1d48efe8d72a4dfebb62a9559e3f4222: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:34.273347 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 81cf492c7a7940a7bf625ec8e2798f29: could not find any suitable replica to move from server 184c543cb05d49cc98ba1c56617be7a5 to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:34.273528 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f69a9daaaa3f44c28eac3e5c9d3b76d5: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:34.273640 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f62a9f6fd824e2e9bb6bbba182f6f56: could not find any suitable replica to move from server b8f581390da449ccb9e299011296405e to server 31f41542452b4b1eb994ff4223d039fc
W20240503 08:12:34.273757 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7f74723fa74a43f590ce741d046208a5: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 31f41542452b4b1eb994ff4223d039fc
W20240503 08:12:34.273878 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 10478ffebe664fcda896887c04a61afa: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:34.273998 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9653225381b14c348f664a692ff48da1: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server 31f41542452b4b1eb994ff4223d039fc
W20240503 08:12:34.274127 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37a3ac4b4467489788ff2543c2b3a179: could not find any suitable replica to move from server 184c543cb05d49cc98ba1c56617be7a5 to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:34.274339 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 19626a36f6c54bd7b459b0de14aadbfc: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:34.274437 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1c93a14622e34eecb8d6bf508be45494: could not find any suitable replica to move from server 184c543cb05d49cc98ba1c56617be7a5 to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:34.274535 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c9883eb3c54f4eb89c079762678d5d4a: could not find any suitable replica to move from server b8f581390da449ccb9e299011296405e to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:34.274664 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f69a9daaaa3f44c28eac3e5c9d3b76d5: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:34.274796 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b3d2ac4623d84f8faf0d7dbe7b38d679: could not find any suitable replica to move from server 184c543cb05d49cc98ba1c56617be7a5 to server 00b61625d8a7460892c118a2570afda5
I20240503 08:12:34.453001 5022 rebalancer_tool.cc:1074] tablet 358cc7bd631d432385d6205db86cbfd4: '184c543cb05d49cc98ba1c56617be7a5' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:12:34.501948 5022 rebalancer_tool.cc:1074] tablet 89d9b1b3c23241d49bfcb914a5bd1777: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:34.801424 5022 rebalancer_tool.cc:1074] tablet 64973388b1c8466798b93ee424df489d: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:12:35.099161 5022 rebalancer_tool.cc:1074] tablet 820ecfef09a34327a08f7c3a893ccb05: '00b61625d8a7460892c118a2570afda5' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:12:35.398988 5022 rebalancer_tool.cc:1074] tablet a752938a2b6c424db0c08bb85b64429b: '184c543cb05d49cc98ba1c56617be7a5' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:12:35.503067 5022 rebalancer_tool.cc:1074] tablet 3219159f92774adbb7ecaa77ac267021: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:35.747689 5022 rebalancer_tool.cc:1074] tablet 7c37418ac97343d1931a8e1239218a38: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:12:35.750546 5022 rebalancer_tool.cc:1084] tablet c00305bb59f84487b2afcdc30910a663: 'b8f581390da449ccb9e299011296405e' -> '00b61625d8a7460892c118a2570afda5' move ignored: Not found: c00305bb59f84487b2afcdc30910a663: tablet not found
I20240503 08:12:35.750731 5022 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 08:12:37.354319 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 81cf492c7a7940a7bf625ec8e2798f29: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:37.354528 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9653225381b14c348f664a692ff48da1: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:37.354593 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1d48efe8d72a4dfebb62a9559e3f4222: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:37.354687 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bc178b718fe049ae96bd1ff6f9794fba: could not find any suitable replica to move from server f24648c92c384f1ba349eee801f985e2 to server 184c543cb05d49cc98ba1c56617be7a5
W20240503 08:12:37.354753 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dde4f7cfffd44f36873e4e6dd5f3a917: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server 31f41542452b4b1eb994ff4223d039fc
W20240503 08:12:37.354866 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ab330af65d474e5b9c89cd53a7f6a8b2: could not find any suitable replica to move from server 184c543cb05d49cc98ba1c56617be7a5 to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:37.354956 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c9883eb3c54f4eb89c079762678d5d4a: could not find any suitable replica to move from server b8f581390da449ccb9e299011296405e to server 5f95b05d111e4090bbc59dec394ed0ab
W20240503 08:12:37.355208 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f69a9daaaa3f44c28eac3e5c9d3b76d5: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:37.355435 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 10478ffebe664fcda896887c04a61afa: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server f24648c92c384f1ba349eee801f985e2
W20240503 08:12:37.355564 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37a3ac4b4467489788ff2543c2b3a179: could not find any suitable replica to move from server b8f581390da449ccb9e299011296405e to server 31f41542452b4b1eb994ff4223d039fc
W20240503 08:12:37.355702 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 19626a36f6c54bd7b459b0de14aadbfc: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:37.355806 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4e422caa613e4bae95fb8412d4e942ca: could not find any suitable replica to move from server b8f581390da449ccb9e299011296405e to server 31f41542452b4b1eb994ff4223d039fc
W20240503 08:12:37.355958 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1d48efe8d72a4dfebb62a9559e3f4222: could not find any suitable replica to move from server 184c543cb05d49cc98ba1c56617be7a5 to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:37.356048 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f69a9daaaa3f44c28eac3e5c9d3b76d5: could not find any suitable replica to move from server 43e2e01cdd314f9c872c4336ef74250a to server f24648c92c384f1ba349eee801f985e2
I20240503 08:12:37.389442 5022 rebalancer_tool.cc:1074] tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83: '00b61625d8a7460892c118a2570afda5' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move scheduled
I20240503 08:12:37.418203 5022 rebalancer_tool.cc:1074] tablet 3cda734f360c4d3eac88a6517028e49d: 'b8f581390da449ccb9e299011296405e' -> '43e2e01cdd314f9c872c4336ef74250a' move scheduled
I20240503 08:12:37.487041 5022 rebalancer_tool.cc:1074] tablet f6cff57835b846e6b48c08bef52f5b68: 'f24648c92c384f1ba349eee801f985e2' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:37.643936 5022 rebalancer_tool.cc:1074] tablet a92cb30e806b4e58a2d9f9cfa516bdf9: '184c543cb05d49cc98ba1c56617be7a5' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:12:37.701018 5022 rebalancer_tool.cc:1074] tablet 733bed28775e4779b74e92d8a395234f: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:12:37.814715 5022 rebalancer_tool.cc:1074] tablet f4900ffdbc624376b32f054efdb100b5: '43e2e01cdd314f9c872c4336ef74250a' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:38.200963 5022 rebalancer_tool.cc:1074] tablet 266d0c75717b4fc98eae8b75a5ba4d8c: '184c543cb05d49cc98ba1c56617be7a5' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:12:38.862039 5022 rebalancer_tool.cc:1136] tablet 7c37418ac97343d1931a8e1239218a38: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:12:38.874079 5022 rebalancer_tool.cc:1136] tablet 868bcace806f4d32977b21c876c2362b: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '00b61625d8a7460892c118a2570afda5' move completed: OK
I20240503 08:12:38.875663 5022 rebalancer_tool.cc:1123] tablet d3b2ebb6eee649199297eb13faa9e4cb: 'b8f581390da449ccb9e299011296405e' -> 'f24648c92c384f1ba349eee801f985e2' move is abandoned: Not found: d3b2ebb6eee649199297eb13faa9e4cb: tablet not found
I20240503 08:12:38.896449 5022 rebalancer_tool.cc:1136] tablet 358cc7bd631d432385d6205db86cbfd4: '184c543cb05d49cc98ba1c56617be7a5' -> 'f24648c92c384f1ba349eee801f985e2' move completed: OK
I20240503 08:12:38.907871 5022 rebalancer_tool.cc:1136] tablet 64973388b1c8466798b93ee424df489d: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:12:38.960167 5022 rebalancer_tool.cc:1136] tablet 48f411ec0bfc4a318b661441c49bd6ec: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:39.079373 5022 rebalancer_tool.cc:1136] tablet 820ecfef09a34327a08f7c3a893ccb05: '00b61625d8a7460892c118a2570afda5' -> 'b8f581390da449ccb9e299011296405e' move completed: OK
I20240503 08:12:39.575023 5022 rebalancer_tool.cc:1136] tablet 89d9b1b3c23241d49bfcb914a5bd1777: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:39.939663 5022 rebalancer_tool.cc:1136] tablet a752938a2b6c424db0c08bb85b64429b: '184c543cb05d49cc98ba1c56617be7a5' -> 'f24648c92c384f1ba349eee801f985e2' move completed: OK
I20240503 08:12:39.976470 5022 rebalancer_tool.cc:1136] tablet 3cda734f360c4d3eac88a6517028e49d: 'b8f581390da449ccb9e299011296405e' -> '43e2e01cdd314f9c872c4336ef74250a' move completed: OK
I20240503 08:12:39.989985 5022 rebalancer_tool.cc:1136] tablet 3219159f92774adbb7ecaa77ac267021: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:40.057344 5022 rebalancer_tool.cc:1074] tablet 314d235cdca241f8b80a7046fb1f524c: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:12:40.159679 5022 rebalancer_tool.cc:1074] tablet 54fe406b5f324d629243f7f939535620: '00b61625d8a7460892c118a2570afda5' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:12:40.311461 5022 rebalancer_tool.cc:1074] tablet 044d8e5a69444865980ffc06a6be9080: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:12:40.415341 5022 rebalancer_tool.cc:1074] tablet f161c8da3b0e4b12b85ca0420a86dc07: '5f95b05d111e4090bbc59dec394ed0ab' -> '43e2e01cdd314f9c872c4336ef74250a' move scheduled
I20240503 08:12:40.524550 5022 rebalancer_tool.cc:1074] tablet 133911d44b394e24a89a5d9443230d32: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:12:40.695755 5022 rebalancer_tool.cc:1074] tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:12:41.037060 5022 rebalancer_tool.cc:1074] tablet 8fc8279b96f74e879b499b0143502c8f: '43e2e01cdd314f9c872c4336ef74250a' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:41.347411 5022 rebalancer_tool.cc:1074] tablet 913a3aacff964ac9a321fac187df33ee: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:12:41.699955 5022 rebalancer_tool.cc:1074] tablet 79579b65538d47f7973fa51f973b3820: '31f41542452b4b1eb994ff4223d039fc' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:41.994966 5022 rebalancer_tool.cc:1074] tablet 53197deeb8d7439c90b019a6a90857e0: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:12:42.618227 5022 rebalancer_tool.cc:1136] tablet f161c8da3b0e4b12b85ca0420a86dc07: '5f95b05d111e4090bbc59dec394ed0ab' -> '43e2e01cdd314f9c872c4336ef74250a' move completed: OK
I20240503 08:12:43.081167 5022 rebalancer_tool.cc:1136] tablet 314d235cdca241f8b80a7046fb1f524c: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:12:43.139158 5022 rebalancer_tool.cc:1136] tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move completed: OK
I20240503 08:12:43.194370 5022 rebalancer_tool.cc:1136] tablet 266d0c75717b4fc98eae8b75a5ba4d8c: '184c543cb05d49cc98ba1c56617be7a5' -> 'f24648c92c384f1ba349eee801f985e2' move completed: OK
I20240503 08:12:43.220666 5022 rebalancer_tool.cc:1136] tablet 8fc8279b96f74e879b499b0143502c8f: '43e2e01cdd314f9c872c4336ef74250a' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:43.233734 5022 rebalancer_tool.cc:1136] tablet 133911d44b394e24a89a5d9443230d32: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:12:43.252383 5022 rebalancer_tool.cc:1136] tablet f4900ffdbc624376b32f054efdb100b5: '43e2e01cdd314f9c872c4336ef74250a' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:43.330963 5022 rebalancer_tool.cc:1136] tablet 1811bd652d2c4afb9f52bcd64ceb2acb: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:12:43.660238 5022 rebalancer_tool.cc:1136] tablet 54fe406b5f324d629243f7f939535620: '00b61625d8a7460892c118a2570afda5' -> '184c543cb05d49cc98ba1c56617be7a5' move completed: OK
I20240503 08:12:43.850993 5022 rebalancer_tool.cc:1136] tablet a92cb30e806b4e58a2d9f9cfa516bdf9: '184c543cb05d49cc98ba1c56617be7a5' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:12:43.936600 5022 rebalancer_tool.cc:1136] tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83: '00b61625d8a7460892c118a2570afda5' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move completed: OK
I20240503 08:12:43.949200 5022 rebalancer_tool.cc:1136] tablet f6cff57835b846e6b48c08bef52f5b68: 'f24648c92c384f1ba349eee801f985e2' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:44.011372 5022 rebalancer_tool.cc:1074] tablet fede923140fb45a5a4f93c0846b62595: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:12:44.130270 5022 rebalancer_tool.cc:1074] tablet 9e371231b34f45dfac455bbbbe156263: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:44.277642 5022 rebalancer_tool.cc:1074] tablet d6658d7bcd634abd808637e474018965: '43e2e01cdd314f9c872c4336ef74250a' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move scheduled
I20240503 08:12:44.384439 5022 rebalancer_tool.cc:1074] tablet fa111e798e3f46d2a5bc1eebfe28f8bf: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:12:44.436156 5022 rebalancer_tool.cc:1074] tablet dab60fefb4704ef19490fb3c76bf47ab: 'f24648c92c384f1ba349eee801f985e2' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:12:44.572027 5022 rebalancer_tool.cc:1074] tablet fba1e1952da44c0e956be1ef7f592f08: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:44.802847 5022 rebalancer_tool.cc:1074] tablet f3123fdbb5c945bd84f97fb4c35c5523: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:12:44.919517 5022 rebalancer_tool.cc:1074] tablet 9cb8a87410b047548efdd973bbc86714: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:45.693302 5022 rebalancer_tool.cc:1136] tablet fede923140fb45a5a4f93c0846b62595: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:12:45.953421 5022 rebalancer_tool.cc:1136] tablet 53197deeb8d7439c90b019a6a90857e0: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move completed: OK
I20240503 08:12:46.282716 5022 rebalancer_tool.cc:1136] tablet 79579b65538d47f7973fa51f973b3820: '31f41542452b4b1eb994ff4223d039fc' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:46.392885 5022 rebalancer_tool.cc:1136] tablet 044d8e5a69444865980ffc06a6be9080: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> 'f24648c92c384f1ba349eee801f985e2' move completed: OK
I20240503 08:12:46.620944 5022 rebalancer_tool.cc:1136] tablet 913a3aacff964ac9a321fac187df33ee: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> 'f24648c92c384f1ba349eee801f985e2' move completed: OK
I20240503 08:12:46.850158 5022 rebalancer_tool.cc:1136] tablet fba1e1952da44c0e956be1ef7f592f08: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:46.866984 5022 rebalancer_tool.cc:1136] tablet 733bed28775e4779b74e92d8a395234f: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '00b61625d8a7460892c118a2570afda5' move completed: OK
I20240503 08:12:47.128314 5022 rebalancer_tool.cc:1074] tablet dc342fed6c6343a080ae0ff2c53cb140: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:12:47.372586 5022 rebalancer_tool.cc:1074] tablet fc4d5ddaaa27463cba31eed19c4c1be0: '184c543cb05d49cc98ba1c56617be7a5' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:12:47.423540 5022 rebalancer_tool.cc:1074] tablet 8737ef0936c641ccaf0b804b1d9c3e07: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:47.563297 5022 rebalancer_tool.cc:1074] tablet b00417d7369a46aabfa933109c645c2c: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:12:47.769865 5022 rebalancer_tool.cc:1136] tablet 9cb8a87410b047548efdd973bbc86714: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:47.865048 5022 rebalancer_tool.cc:1136] tablet dab60fefb4704ef19490fb3c76bf47ab: 'f24648c92c384f1ba349eee801f985e2' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:12:47.890870 5022 rebalancer_tool.cc:1136] tablet fa111e798e3f46d2a5bc1eebfe28f8bf: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move completed: OK
I20240503 08:12:48.048820 5022 rebalancer_tool.cc:1136] tablet d6658d7bcd634abd808637e474018965: '43e2e01cdd314f9c872c4336ef74250a' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move completed: OK
I20240503 08:12:48.167615 5022 rebalancer_tool.cc:1136] tablet 9e371231b34f45dfac455bbbbe156263: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:48.310760 5022 rebalancer_tool.cc:1136] tablet f3123fdbb5c945bd84f97fb4c35c5523: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '00b61625d8a7460892c118a2570afda5' move completed: OK
I20240503 08:12:48.398415 5022 rebalancer_tool.cc:1074] tablet 9d11cd50b562408f9d3bc24a974bdba8: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:12:48.450134 5022 rebalancer_tool.cc:1074] tablet 674e558bf5db4f169f8cdcb08c40f5fb: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:48.489122 5022 rebalancer_tool.cc:1074] tablet 1d4f6c5975f8446a9f6ba8cae137bda0: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:12:48.857939 5022 rebalancer_tool.cc:1136] tablet dc342fed6c6343a080ae0ff2c53cb140: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:12:49.536924 5022 rebalancer_tool.cc:1136] tablet fc4d5ddaaa27463cba31eed19c4c1be0: '184c543cb05d49cc98ba1c56617be7a5' -> 'f24648c92c384f1ba349eee801f985e2' move completed: OK
I20240503 08:12:50.389130 5022 rebalancer_tool.cc:1136] tablet 8737ef0936c641ccaf0b804b1d9c3e07: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:50.678663 5022 rebalancer_tool.cc:1136] tablet b00417d7369a46aabfa933109c645c2c: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> 'f24648c92c384f1ba349eee801f985e2' move completed: OK
I20240503 08:12:50.722892 5022 rebalancer_tool.cc:1136] tablet 1d4f6c5975f8446a9f6ba8cae137bda0: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '184c543cb05d49cc98ba1c56617be7a5' move completed: OK
I20240503 08:12:50.765300 5022 rebalancer_tool.cc:1136] tablet 674e558bf5db4f169f8cdcb08c40f5fb: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:50.782490 5022 rebalancer_tool.cc:1136] tablet 9d11cd50b562408f9d3bc24a974bdba8: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move completed: OK
W20240503 08:12:51.842761 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4d2142b44bc4afa85d24f2a4c14b381: could not find any suitable replica to move from server f24648c92c384f1ba349eee801f985e2 to server 315f3f0bcbf447a4b93b1aafcfbf0f5a
W20240503 08:12:51.843104 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 90ccd16da234473f86755bf3c2e4f43f: could not find any suitable replica to move from server f24648c92c384f1ba349eee801f985e2 to server 00b61625d8a7460892c118a2570afda5
W20240503 08:12:51.843291 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8a49b8cedbec4f2a8efa8bb7c5a902cd: could not find any suitable replica to move from server 315f3f0bcbf447a4b93b1aafcfbf0f5a to server 184c543cb05d49cc98ba1c56617be7a5
I20240503 08:12:51.885799 5022 rebalancer_tool.cc:1074] tablet b48714749469474886f9e1d7e7635915: '31f41542452b4b1eb994ff4223d039fc' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:12:51.916606 5022 rebalancer_tool.cc:1074] tablet 53ce7a6567274050a2b377a9c650ccc4: '5f95b05d111e4090bbc59dec394ed0ab' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:12:51.968876 5022 rebalancer_tool.cc:1074] tablet 40476d1762154f6d9ce4855872492573: '00b61625d8a7460892c118a2570afda5' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:12:52.012795 5022 rebalancer_tool.cc:1074] tablet 54fe406b5f324d629243f7f939535620: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:12:52.119581 5022 rebalancer_tool.cc:1074] tablet 7d1e2530d5f2431baf0b2dd41081bb9d: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:12:52.328539 5022 rebalancer_tool.cc:1074] tablet 34b7fd8dc7434b8ea14b9deb0023b4cb: '5f95b05d111e4090bbc59dec394ed0ab' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:12:52.446163 5022 rebalancer_tool.cc:1074] tablet 84831077defe434c9f29eecfae7b44c8: '00b61625d8a7460892c118a2570afda5' -> '43e2e01cdd314f9c872c4336ef74250a' move scheduled
I20240503 08:12:52.871050 5022 rebalancer_tool.cc:1074] tablet 275fc5bbb1494306b1139b7f950ef80f: 'f24648c92c384f1ba349eee801f985e2' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move scheduled
I20240503 08:12:53.139041 5022 rebalancer_tool.cc:1074] tablet 47dde43816964e89bfb235aad34e9583: '31f41542452b4b1eb994ff4223d039fc' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:12:53.490486 5022 rebalancer_tool.cc:1074] tablet 53197deeb8d7439c90b019a6a90857e0: 'b8f581390da449ccb9e299011296405e' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:12:54.296471 5022 rebalancer_tool.cc:1074] tablet 3219159f92774adbb7ecaa77ac267021: '43e2e01cdd314f9c872c4336ef74250a' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:12:54.640897 5022 rebalancer_tool.cc:1074] tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:12:54.904138 5022 rebalancer_tool.cc:1074] tablet 9a0bffb0aa304bd184b8ad26bc94fd18: '31f41542452b4b1eb994ff4223d039fc' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:12:55.133116 5022 rebalancer_tool.cc:1074] tablet 0caddf5e39f8440baa8ad3656ce91a8e: 'b8f581390da449ccb9e299011296405e' -> '43e2e01cdd314f9c872c4336ef74250a' move scheduled
I20240503 08:12:55.329321 5022 rebalancer_tool.cc:1074] tablet 1e641fe703c14f4cb1bd22c676d79576: '5f95b05d111e4090bbc59dec394ed0ab' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move scheduled
I20240503 08:12:55.484297 5022 rebalancer_tool.cc:1074] tablet df06752e8d7247dd962a376c629473c3: '00b61625d8a7460892c118a2570afda5' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:12:55.791038 5022 rebalancer_tool.cc:1074] tablet 09a268c672ba4580a2d93b5d15d12666: 'f24648c92c384f1ba349eee801f985e2' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:12:56.131588 5022 rebalancer_tool.cc:1074] tablet 929d8d354b914502b5e677ccc84e2eda: '43e2e01cdd314f9c872c4336ef74250a' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:12:56.546948 5022 rebalancer_tool.cc:1074] tablet 98ad826dca7c4343afcae34bf1a40da8: '5f95b05d111e4090bbc59dec394ed0ab' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move scheduled
I20240503 08:12:57.546717 5022 rebalancer_tool.cc:1136] tablet df06752e8d7247dd962a376c629473c3: '00b61625d8a7460892c118a2570afda5' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:12:57.663903 5022 rebalancer_tool.cc:1136] tablet 1e641fe703c14f4cb1bd22c676d79576: '5f95b05d111e4090bbc59dec394ed0ab' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move completed: OK
I20240503 08:12:57.807917 5022 rebalancer_tool.cc:1136] tablet 0caddf5e39f8440baa8ad3656ce91a8e: 'b8f581390da449ccb9e299011296405e' -> '43e2e01cdd314f9c872c4336ef74250a' move completed: OK
W20240503 08:12:58.313853 5061 kernel_stack_watchdog.cc:198] Thread 5022 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 385ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe649ac9980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7fe646737d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7fe64d38f530 kudu::rpc::ReactorThread::Init() at ??:0
@ 0x7fe64d39a797 kudu::rpc::Reactor::Init() at ??:0
@ 0x7fe64d3435c5 kudu::rpc::Messenger::Init() at ??:0
@ 0x7fe64d3429d1 kudu::rpc::MessengerBuilder::Build() at ??:0
@ 0x7fe65a61fbb2 kudu::tools::BuildMessenger() at ??:0
@ 0x7fe65a6399c8 kudu::tools::BuildProxy<>() at ??:0
@ 0x7fe65b5592a9 kudu::tools::CheckCompleteMove() at ??:0
@ 0x7fe65b4df1df kudu::tools::RebalancerTool::AlgoBasedRunner::UpdateMovesInProgressStatus() at ??:0
@ 0x7fe65b4d682b kudu::tools::RebalancerTool::RunWith() at ??:0
@ 0x7fe65b4d314c kudu::tools::RebalancerTool::Run() at ??:0
@ 0x97996b kudu::tools::(anonymous namespace)::RunRebalance() at ??:?
W20240503 08:12:58.350770 5022 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 0.523s user 0.175s sys 0.218s
W20240503 08:12:58.351565 5022 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 0.523s user 0.175s sys 0.218s
I20240503 08:12:58.394367 5022 rebalancer_tool.cc:1136] tablet 9a0bffb0aa304bd184b8ad26bc94fd18: '31f41542452b4b1eb994ff4223d039fc' -> '184c543cb05d49cc98ba1c56617be7a5' move completed: OK
I20240503 08:12:59.056372 5022 rebalancer_tool.cc:1136] tablet 3219159f92774adbb7ecaa77ac267021: '43e2e01cdd314f9c872c4336ef74250a' -> '00b61625d8a7460892c118a2570afda5' move completed: OK
I20240503 08:12:59.094049 5022 rebalancer_tool.cc:1136] tablet 53197deeb8d7439c90b019a6a90857e0: 'b8f581390da449ccb9e299011296405e' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:12:59.140192 5022 rebalancer_tool.cc:1136] tablet 47dde43816964e89bfb235aad34e9583: '31f41542452b4b1eb994ff4223d039fc' -> '184c543cb05d49cc98ba1c56617be7a5' move completed: OK
I20240503 08:12:59.152832 5022 rebalancer_tool.cc:1136] tablet 84831077defe434c9f29eecfae7b44c8: '00b61625d8a7460892c118a2570afda5' -> '43e2e01cdd314f9c872c4336ef74250a' move completed: OK
I20240503 08:12:59.169592 5022 rebalancer_tool.cc:1136] tablet 275fc5bbb1494306b1139b7f950ef80f: 'f24648c92c384f1ba349eee801f985e2' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move completed: OK
I20240503 08:12:59.182083 5022 rebalancer_tool.cc:1136] tablet 34b7fd8dc7434b8ea14b9deb0023b4cb: '5f95b05d111e4090bbc59dec394ed0ab' -> 'b8f581390da449ccb9e299011296405e' move completed: OK
I20240503 08:12:59.197613 5022 rebalancer_tool.cc:1136] tablet 7d1e2530d5f2431baf0b2dd41081bb9d: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> '184c543cb05d49cc98ba1c56617be7a5' move completed: OK
I20240503 08:12:59.210243 5022 rebalancer_tool.cc:1136] tablet 54fe406b5f324d629243f7f939535620: '43e2e01cdd314f9c872c4336ef74250a' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:12:59.223313 5022 rebalancer_tool.cc:1136] tablet 40476d1762154f6d9ce4855872492573: '00b61625d8a7460892c118a2570afda5' -> 'f24648c92c384f1ba349eee801f985e2' move completed: OK
I20240503 08:12:59.250567 5022 rebalancer_tool.cc:1136] tablet 53ce7a6567274050a2b377a9c650ccc4: '5f95b05d111e4090bbc59dec394ed0ab' -> 'b8f581390da449ccb9e299011296405e' move completed: OK
I20240503 08:12:59.326301 5022 rebalancer_tool.cc:1136] tablet b48714749469474886f9e1d7e7635915: '31f41542452b4b1eb994ff4223d039fc' -> '184c543cb05d49cc98ba1c56617be7a5' move completed: OK
I20240503 08:12:59.410339 5022 rebalancer_tool.cc:1074] tablet f4a205bf277d4770836dbbf0ec28d884: '00b61625d8a7460892c118a2570afda5' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:12:59.648304 5022 rebalancer_tool.cc:1074] tablet df0c416025c74075b6429d169fcdd58c: '31f41542452b4b1eb994ff4223d039fc' -> '43e2e01cdd314f9c872c4336ef74250a' move scheduled
I20240503 08:12:59.820223 5022 rebalancer_tool.cc:1074] tablet 01821d71873d4ad78d7b7f685161a229: '5f95b05d111e4090bbc59dec394ed0ab' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:12:59.891042 5022 rebalancer_tool.cc:1074] tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f: '00b61625d8a7460892c118a2570afda5' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move scheduled
I20240503 08:12:59.952490 5022 rebalancer_tool.cc:1074] tablet fe20c189635648f5b78905e8b9643bcd: '31f41542452b4b1eb994ff4223d039fc' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:13:00.056560 5022 rebalancer_tool.cc:1074] tablet c6994370a532472893cc934337f3cb6e: 'f24648c92c384f1ba349eee801f985e2' -> '43e2e01cdd314f9c872c4336ef74250a' move scheduled
I20240503 08:13:00.112480 5022 rebalancer_tool.cc:1074] tablet dab60fefb4704ef19490fb3c76bf47ab: '5f95b05d111e4090bbc59dec394ed0ab' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:13:00.306846 5022 rebalancer_tool.cc:1074] tablet fede923140fb45a5a4f93c0846b62595: 'b8f581390da449ccb9e299011296405e' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:13:00.412393 5022 rebalancer_tool.cc:1074] tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83: '31f41542452b4b1eb994ff4223d039fc' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:13:00.515646 5022 rebalancer_tool.cc:1074] tablet a92cb30e806b4e58a2d9f9cfa516bdf9: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:13:00.617902 5022 rebalancer_tool.cc:1074] tablet eb4e33ffc60043d9a50f8381e85b927a: '5f95b05d111e4090bbc59dec394ed0ab' -> '43e2e01cdd314f9c872c4336ef74250a' move scheduled
I20240503 08:13:00.696849 5022 rebalancer_tool.cc:1074] tablet 044d8e5a69444865980ffc06a6be9080: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:13:00.750352 5022 rebalancer_tool.cc:1074] tablet 913a3aacff964ac9a321fac187df33ee: 'b8f581390da449ccb9e299011296405e' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:13:00.852540 5022 rebalancer_tool.cc:1074] tablet 437037e814464cf5959153bcd3efc145: '5f95b05d111e4090bbc59dec394ed0ab' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move scheduled
I20240503 08:13:02.352397 5022 rebalancer_tool.cc:1136] tablet dab60fefb4704ef19490fb3c76bf47ab: '5f95b05d111e4090bbc59dec394ed0ab' -> '184c543cb05d49cc98ba1c56617be7a5' move completed: OK
I20240503 08:13:02.519225 5022 rebalancer_tool.cc:1136] tablet fe20c189635648f5b78905e8b9643bcd: '31f41542452b4b1eb994ff4223d039fc' -> 'b8f581390da449ccb9e299011296405e' move completed: OK
I20240503 08:13:02.532308 5022 rebalancer_tool.cc:1136] tablet 4d3aaa1025ca4b0e832535d1ca3d5c4f: '00b61625d8a7460892c118a2570afda5' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move completed: OK
I20240503 08:13:02.544754 5022 rebalancer_tool.cc:1136] tablet 01821d71873d4ad78d7b7f685161a229: '5f95b05d111e4090bbc59dec394ed0ab' -> '184c543cb05d49cc98ba1c56617be7a5' move completed: OK
I20240503 08:13:02.560472 5022 rebalancer_tool.cc:1136] tablet df0c416025c74075b6429d169fcdd58c: '31f41542452b4b1eb994ff4223d039fc' -> '43e2e01cdd314f9c872c4336ef74250a' move completed: OK
I20240503 08:13:02.571480 5022 rebalancer_tool.cc:1136] tablet f4a205bf277d4770836dbbf0ec28d884: '00b61625d8a7460892c118a2570afda5' -> 'b8f581390da449ccb9e299011296405e' move completed: OK
I20240503 08:13:02.583086 5022 rebalancer_tool.cc:1136] tablet 98ad826dca7c4343afcae34bf1a40da8: '5f95b05d111e4090bbc59dec394ed0ab' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move completed: OK
I20240503 08:13:02.595836 5022 rebalancer_tool.cc:1136] tablet 929d8d354b914502b5e677ccc84e2eda: '43e2e01cdd314f9c872c4336ef74250a' -> 'b8f581390da449ccb9e299011296405e' move completed: OK
I20240503 08:13:02.663496 5022 rebalancer_tool.cc:1136] tablet 09a268c672ba4580a2d93b5d15d12666: 'f24648c92c384f1ba349eee801f985e2' -> '184c543cb05d49cc98ba1c56617be7a5' move completed: OK
I20240503 08:13:02.718274 5022 rebalancer_tool.cc:1136] tablet 1cf9b2eef42e4fff8cdf5f9dd7b8b31e: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> 'f24648c92c384f1ba349eee801f985e2' move completed: OK
I20240503 08:13:02.771338 5022 rebalancer_tool.cc:1074] tablet f3123fdbb5c945bd84f97fb4c35c5523: 'b8f581390da449ccb9e299011296405e' -> '31f41542452b4b1eb994ff4223d039fc' move scheduled
I20240503 08:13:02.863128 5022 rebalancer_tool.cc:1074] tablet 4422e4f265f344039734893b759c2050: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:13:02.957232 5022 rebalancer_tool.cc:1074] tablet ebd7877c8faf409a8d042bef38f218aa: '5f95b05d111e4090bbc59dec394ed0ab' -> '43e2e01cdd314f9c872c4336ef74250a' move scheduled
I20240503 08:13:03.069137 5022 rebalancer_tool.cc:1074] tablet c4177c11467b49d49452ac10bd983627: 'f24648c92c384f1ba349eee801f985e2' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:13:03.128733 5022 rebalancer_tool.cc:1074] tablet 733bed28775e4779b74e92d8a395234f: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:13:03.175992 5022 rebalancer_tool.cc:1074] tablet 674e558bf5db4f169f8cdcb08c40f5fb: '43e2e01cdd314f9c872c4336ef74250a' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:13:03.335438 5022 rebalancer_tool.cc:1074] tablet 7c37418ac97343d1931a8e1239218a38: '184c543cb05d49cc98ba1c56617be7a5' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:13:03.625833 5022 rebalancer_tool.cc:1136] tablet 437037e814464cf5959153bcd3efc145: '5f95b05d111e4090bbc59dec394ed0ab' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move completed: OK
I20240503 08:13:03.688277 5022 rebalancer_tool.cc:1136] tablet 044d8e5a69444865980ffc06a6be9080: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move completed: OK
I20240503 08:13:03.734411 5022 rebalancer_tool.cc:1136] tablet eb4e33ffc60043d9a50f8381e85b927a: '5f95b05d111e4090bbc59dec394ed0ab' -> '43e2e01cdd314f9c872c4336ef74250a' move completed: OK
I20240503 08:13:03.831410 5022 rebalancer_tool.cc:1136] tablet a92cb30e806b4e58a2d9f9cfa516bdf9: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> 'f24648c92c384f1ba349eee801f985e2' move completed: OK
I20240503 08:13:03.893560 5022 rebalancer_tool.cc:1136] tablet 2d4d06fdfc4e49d5b0790e2ee3b4cc83: '31f41542452b4b1eb994ff4223d039fc' -> '184c543cb05d49cc98ba1c56617be7a5' move completed: OK
I20240503 08:13:03.973912 5022 rebalancer_tool.cc:1136] tablet c6994370a532472893cc934337f3cb6e: 'f24648c92c384f1ba349eee801f985e2' -> '43e2e01cdd314f9c872c4336ef74250a' move completed: OK
I20240503 08:13:04.046072 5022 rebalancer_tool.cc:1136] tablet fede923140fb45a5a4f93c0846b62595: 'b8f581390da449ccb9e299011296405e' -> '00b61625d8a7460892c118a2570afda5' move completed: OK
I20240503 08:13:04.105564 5022 rebalancer_tool.cc:1074] tablet 9c02a6bb7f7d4b42a6af774f342b0398: 'f24648c92c384f1ba349eee801f985e2' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:13:04.636723 5022 rebalancer_tool.cc:1136] tablet 4422e4f265f344039734893b759c2050: '184c543cb05d49cc98ba1c56617be7a5' -> '00b61625d8a7460892c118a2570afda5' move completed: OK
I20240503 08:13:04.759290 5022 rebalancer_tool.cc:1136] tablet f3123fdbb5c945bd84f97fb4c35c5523: 'b8f581390da449ccb9e299011296405e' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:13:04.801460 5022 rebalancer_tool.cc:1136] tablet 913a3aacff964ac9a321fac187df33ee: 'b8f581390da449ccb9e299011296405e' -> '31f41542452b4b1eb994ff4223d039fc' move completed: OK
I20240503 08:13:04.967552 5022 rebalancer_tool.cc:1074] tablet 6d69a5126dff43af8ba3acc931d17799: 'b8f581390da449ccb9e299011296405e' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:13:05.030495 5022 rebalancer_tool.cc:1074] tablet 415a75c852814ea89b65d294d682c571: 'f24648c92c384f1ba349eee801f985e2' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:13:05.102819 5022 rebalancer_tool.cc:1136] tablet 7c37418ac97343d1931a8e1239218a38: '184c543cb05d49cc98ba1c56617be7a5' -> 'f24648c92c384f1ba349eee801f985e2' move completed: OK
I20240503 08:13:05.135383 5022 rebalancer_tool.cc:1136] tablet c4177c11467b49d49452ac10bd983627: 'f24648c92c384f1ba349eee801f985e2' -> 'b8f581390da449ccb9e299011296405e' move completed: OK
I20240503 08:13:05.160436 5022 rebalancer_tool.cc:1136] tablet ebd7877c8faf409a8d042bef38f218aa: '5f95b05d111e4090bbc59dec394ed0ab' -> '43e2e01cdd314f9c872c4336ef74250a' move completed: OK
I20240503 08:13:05.177191 5022 rebalancer_tool.cc:1136] tablet 733bed28775e4779b74e92d8a395234f: '184c543cb05d49cc98ba1c56617be7a5' -> '5f95b05d111e4090bbc59dec394ed0ab' move completed: OK
I20240503 08:13:05.543908 5022 rebalancer_tool.cc:1136] tablet 674e558bf5db4f169f8cdcb08c40f5fb: '43e2e01cdd314f9c872c4336ef74250a' -> 'b8f581390da449ccb9e299011296405e' move completed: OK
I20240503 08:13:06.389055 5022 rebalancer_tool.cc:1136] tablet 9c02a6bb7f7d4b42a6af774f342b0398: 'f24648c92c384f1ba349eee801f985e2' -> 'b8f581390da449ccb9e299011296405e' move completed: OK
I20240503 08:13:06.402243 5022 rebalancer_tool.cc:1136] tablet 6d69a5126dff43af8ba3acc931d17799: 'b8f581390da449ccb9e299011296405e' -> '184c543cb05d49cc98ba1c56617be7a5' move completed: OK
I20240503 08:13:06.731187 5022 rebalancer_tool.cc:1136] tablet 415a75c852814ea89b65d294d682c571: 'f24648c92c384f1ba349eee801f985e2' -> 'b8f581390da449ccb9e299011296405e' move completed: OK
W20240503 08:13:07.523463 5022 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f0aa7c71965d48539c0ac7118319411e: could not find any suitable replica to move from server f24648c92c384f1ba349eee801f985e2 to server 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:07.564880 5022 rebalancer_tool.cc:1074] tablet 5785af2b9ba34a3b95c5ffdb024b1c8f: '00b61625d8a7460892c118a2570afda5' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:13:07.598641 5022 rebalancer_tool.cc:1074] tablet 8f3e931766d44f0aaf902cf0e950f83b: 'b8f581390da449ccb9e299011296405e' -> '43e2e01cdd314f9c872c4336ef74250a' move scheduled
I20240503 08:13:07.671161 5022 rebalancer_tool.cc:1074] tablet a97625d919124c559b1e9a5cb785153f: 'f24648c92c384f1ba349eee801f985e2' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:13:07.728330 5022 rebalancer_tool.cc:1074] tablet a7e0415e028b4303bcf4ab64f4ec6b54: '315f3f0bcbf447a4b93b1aafcfbf0f5a' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:13:07.775590 5022 rebalancer_tool.cc:1074] tablet 826e7d6e6d2947fbb2a178e8e2a32d56: '31f41542452b4b1eb994ff4223d039fc' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:13:07.865803 5022 rebalancer_tool.cc:1074] tablet 0c3070c262ed420a91ac0ed063d16471: '31f41542452b4b1eb994ff4223d039fc' -> '184c543cb05d49cc98ba1c56617be7a5' move scheduled
I20240503 08:13:07.930472 5022 rebalancer_tool.cc:1074] tablet 438146d5fa194156b929d7190b11a5de: '43e2e01cdd314f9c872c4336ef74250a' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move scheduled
I20240503 08:13:07.969482 5022 rebalancer_tool.cc:1074] tablet 12b0c4a0b0564a279806d3fd6f981eed: 'b8f581390da449ccb9e299011296405e' -> 'f24648c92c384f1ba349eee801f985e2' move scheduled
I20240503 08:13:08.032755 5022 rebalancer_tool.cc:1074] tablet 0ba03a430b4b498f9ba66d733d68b13c: '31f41542452b4b1eb994ff4223d039fc' -> '5f95b05d111e4090bbc59dec394ed0ab' move scheduled
I20240503 08:13:08.091451 5022 rebalancer_tool.cc:1074] tablet 7a792a309b3c43f482b4adb0fbcfadde: '00b61625d8a7460892c118a2570afda5' -> '43e2e01cdd314f9c872c4336ef74250a' move scheduled
I20240503 08:13:08.168370 5022 rebalancer_tool.cc:1074] tablet 51f3edb7a72c4ba083429c150fa73363: '5f95b05d111e4090bbc59dec394ed0ab' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move scheduled
I20240503 08:13:08.271358 5022 rebalancer_tool.cc:1074] tablet e613a6cbac2c4ad9b71442baa71ba9a3: 'f24648c92c384f1ba349eee801f985e2' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:13:08.407609 5022 rebalancer_tool.cc:1074] tablet a03edb25e08e495c8dcb46adc9d8d2a4: '43e2e01cdd314f9c872c4336ef74250a' -> 'b8f581390da449ccb9e299011296405e' move scheduled
I20240503 08:13:08.514312 5022 rebalancer_tool.cc:1074] tablet d6b22a007a2540ea9364579ac680fcf4: '31f41542452b4b1eb994ff4223d039fc' -> '00b61625d8a7460892c118a2570afda5' move scheduled
I20240503 08:13:08.752704 5022 rebalancer_tool.cc:1074] tablet c2d33b2dc884448483072fa6ab825f0b: '5f95b05d111e4090bbc59dec394ed0ab' -> 'b8f581390da449ccb9e299011296405e' move scheduled
W20240503 08:13:09.063161 9815 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:13:08.948697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.157.197:42465 (local address 127.0.0.1:43164)
0503 08:13:08.954552 (+ 5855us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:08.954575 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 08:13:08.954714 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 08:13:09.061609 (+106895us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":5702,"thread_start_us":5800,"threads_started":1}
I20240503 08:13:09.064172 5022 rebalancer_tool.cc:1084] tablet 8ae7520f758c47a39cfa3aaf59ebc674: '43e2e01cdd314f9c872c4336ef74250a' -> '315f3f0bcbf447a4b93b1aafcfbf0f5a' move ignored: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:09.064314 5022 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): unacceptable health status UNAVAILABLE
I20240503 08:13:11.808068 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 1731
I20240503 08:13:11.817113 1853 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:13:11.820058 2137 consensus_peers.cc:485] T df06752e8d7247dd962a376c629473c3 P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:11.854986 1877 consensus_peers.cc:462] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:12.081153 1656 catalog_manager.cc:4573] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 51f3edb7a72c4ba083429c150fa73363: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:12.090679 2137 consensus_peers.cc:462] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:12.191154 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 1731
W20240503 08:13:12.252839 2009 connection.cc:535] client connection to 127.27.157.193:41181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.253352 4902 connection.cc:535] client connection to 127.27.157.193:41181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.253751 9822 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:13:12.252953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.157.193:41181 (local address 127.27.157.196:45477)
0503 08:13:12.253483 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:12.253575 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":403}
I20240503 08:13:12.253829 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 1862
W20240503 08:13:12.254072 2010 connection.cc:535] server connection from 127.27.157.193:45249 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.254279 2140 connection.cc:535] server connection from 127.27.157.193:56947 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.254892 2009 consensus_peers.cc:485] T 0c3070c262ed420a91ac0ed063d16471 P 43e2e01cdd314f9c872c4336ef74250a -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.254995 2009 consensus_peers.cc:485] T a752938a2b6c424db0c08bb85b64429b P 43e2e01cdd314f9c872c4336ef74250a -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.255069 2009 consensus_peers.cc:485] T c2d33b2dc884448483072fa6ab825f0b P 43e2e01cdd314f9c872c4336ef74250a -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.255056 4637 connection.cc:535] client connection to 127.27.157.193:41181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.255127 2009 consensus_peers.cc:485] T 938428b201e842448232a1d77061afab P 43e2e01cdd314f9c872c4336ef74250a -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.255183 2009 consensus_peers.cc:485] T 79579b65538d47f7973fa51f973b3820 P 43e2e01cdd314f9c872c4336ef74250a -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.255246 2009 consensus_peers.cc:485] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.255460 2139 consensus_peers.cc:485] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.255597 2139 consensus_peers.cc:485] T c4177c11467b49d49452ac10bd983627 P 184c543cb05d49cc98ba1c56617be7a5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.255677 2139 consensus_peers.cc:485] T 1811bd652d2c4afb9f52bcd64ceb2acb P 184c543cb05d49cc98ba1c56617be7a5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.255739 2139 consensus_peers.cc:485] T dc342fed6c6343a080ae0ff2c53cb140 P 184c543cb05d49cc98ba1c56617be7a5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.255815 4769 connection.cc:535] client connection to 127.27.157.193:41181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.256063 4902 consensus_peers.cc:485] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.256238 4902 consensus_peers.cc:485] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.256350 4902 consensus_peers.cc:485] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.256479 4902 consensus_peers.cc:485] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.256600 4902 consensus_peers.cc:485] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.256819 4637 consensus_peers.cc:485] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.256951 4637 consensus_peers.cc:485] T f6cff57835b846e6b48c08bef52f5b68 P 31f41542452b4b1eb994ff4223d039fc -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.257035 4637 consensus_peers.cc:485] T 53ce7a6567274050a2b377a9c650ccc4 P 31f41542452b4b1eb994ff4223d039fc -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.257123 4637 consensus_peers.cc:485] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.258373 4900 connection.cc:535] server connection from 127.27.157.193:35539 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.259224 4769 consensus_peers.cc:485] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.259483 4769 consensus_peers.cc:485] T d6b22a007a2540ea9364579ac680fcf4 P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.259716 4769 consensus_peers.cc:485] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.259912 4769 consensus_peers.cc:485] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.260066 4769 consensus_peers.cc:485] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.260237 4769 consensus_peers.cc:485] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.265421 1984 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:13:12.286811 4637 consensus_peers.cc:485] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.343750 4902 consensus_peers.cc:485] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.343946 4769 consensus_peers.cc:485] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.426909 4635 consensus_peers.cc:485] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:12.606334 4767 consensus_peers.cc:485] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:12.651405 1877 consensus_peers.cc:462] T a97625d919124c559b1e9a5cb785153f P 315f3f0bcbf447a4b93b1aafcfbf0f5a -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:12.651705 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 1862
W20240503 08:13:12.681262 2137 consensus_peers.cc:462] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:12.729478 4769 connection.cc:535] server connection from 127.27.157.194:55239 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.729558 2007 connection.cc:535] server connection from 127.27.157.194:46923 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.729593 4902 connection.cc:535] server connection from 127.27.157.194:41645 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.729785 4769 connection.cc:535] client connection to 127.27.157.194:41975 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.730202 2139 connection.cc:535] server connection from 127.27.157.194:37123 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.730275 1659 connection.cc:535] server connection from 127.27.157.194:56705 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:12.730494 2009 connection.cc:535] client connection to 127.27.157.194:41975 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:12.730958 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 1992
W20240503 08:13:12.731735 2009 consensus_peers.cc:485] T 48f411ec0bfc4a318b661441c49bd6ec P 43e2e01cdd314f9c872c4336ef74250a -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.732007 4769 consensus_peers.cc:485] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.732128 4769 consensus_peers.cc:485] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.732187 4769 consensus_peers.cc:485] T 05274a8ee1ac4cada1f7788f43d7f1f3 P f24648c92c384f1ba349eee801f985e2 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.732225 4769 consensus_peers.cc:485] T 1e641fe703c14f4cb1bd22c676d79576 P f24648c92c384f1ba349eee801f985e2 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.732223 2139 consensus_peers.cc:485] T fe20c189635648f5b78905e8b9643bcd P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.732264 4769 consensus_peers.cc:485] T d6658d7bcd634abd808637e474018965 P f24648c92c384f1ba349eee801f985e2 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.732548 4902 consensus_peers.cc:485] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.732671 4902 consensus_peers.cc:485] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.738476 2114 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:13:12.744444 4769 consensus_peers.cc:485] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.760032 4902 consensus_peers.cc:485] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.796150 4635 consensus_peers.cc:485] T c4e697bc477a4899b69e4fdcc4e235e7 P 31f41542452b4b1eb994ff4223d039fc -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:12.818379 10038 raft_consensus.cc:482] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:12.818600 10038 raft_consensus.cc:504] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:12.819419 10038 leader_election.cc:288] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 00b61625d8a7460892c118a2570afda5 (127.27.157.200:45283)
I20240503 08:13:12.820591 4964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868bcace806f4d32977b21c876c2362b" candidate_uuid: "184c543cb05d49cc98ba1c56617be7a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "00b61625d8a7460892c118a2570afda5" is_pre_election: true
W20240503 08:13:12.820984 2139 leader_election.cc:334] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
I20240503 08:13:12.821223 2138 leader_election.cc:302] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 184c543cb05d49cc98ba1c56617be7a5; no voters: 00b61625d8a7460892c118a2570afda5, b8f581390da449ccb9e299011296405e
I20240503 08:13:12.821470 10038 raft_consensus.cc:2726] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:13:12.833130 2139 consensus_peers.cc:485] T 314d235cdca241f8b80a7046fb1f524c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.838704 9878 raft_consensus.cc:482] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:12.838869 9878 raft_consensus.cc:504] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:13:12.839555 9878 leader_election.cc:288] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
W20240503 08:13:12.841054 4900 leader_election.cc:334] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:12.841187 4902 leader_election.cc:334] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
I20240503 08:13:12.841291 4902 leader_election.cc:302] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 5f95b05d111e4090bbc59dec394ed0ab, b8f581390da449ccb9e299011296405e
I20240503 08:13:12.841492 9878 raft_consensus.cc:2726] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 08:13:12.906680 4902 consensus_peers.cc:485] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.943346 4637 consensus_peers.cc:485] T 9c02a6bb7f7d4b42a6af774f342b0398 P 31f41542452b4b1eb994ff4223d039fc -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:12.946861 10060 raft_consensus.cc:482] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:12.947060 10060 raft_consensus.cc:504] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:13:12.947820 10060 leader_election.cc:288] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:13:12.948406 4829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e36553b05f0491dbd70a6d7eb12d7f5" candidate_uuid: "31f41542452b4b1eb994ff4223d039fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f24648c92c384f1ba349eee801f985e2" is_pre_election: true
W20240503 08:13:12.949350 4637 leader_election.cc:334] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
I20240503 08:13:12.949468 4637 leader_election.cc:302] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc; no voters: b8f581390da449ccb9e299011296405e, f24648c92c384f1ba349eee801f985e2
I20240503 08:13:12.949712 10060 raft_consensus.cc:2726] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:13:12.985922 4637 consensus_peers.cc:485] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:12.986163 4902 consensus_peers.cc:485] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.092510 2139 consensus_peers.cc:485] T 64973388b1c8466798b93ee424df489d P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.106410 2137 consensus_peers.cc:462] T 0ba03a430b4b498f9ba66d733d68b13c P 184c543cb05d49cc98ba1c56617be7a5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:13.123386 2009 consensus_peers.cc:485] T a7e0415e028b4303bcf4ab64f4ec6b54 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.123589 2009 consensus_peers.cc:485] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P 43e2e01cdd314f9c872c4336ef74250a -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.123682 2009 consensus_peers.cc:485] T 2d4d06fdfc4e49d5b0790e2ee3b4cc83 P 43e2e01cdd314f9c872c4336ef74250a -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.123781 2009 consensus_peers.cc:485] T 266d0c75717b4fc98eae8b75a5ba4d8c P 43e2e01cdd314f9c872c4336ef74250a -> Peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Couldn't send request to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.124215 10057 raft_consensus.cc:482] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:13.124411 10057 raft_consensus.cc:504] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:13.125103 10057 leader_election.cc:288] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:13:13.125603 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868bcace806f4d32977b21c876c2362b" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5" is_pre_election: true
I20240503 08:13:13.125841 2202 raft_consensus.cc:2454] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 1.
I20240503 08:13:13.126335 4901 leader_election.cc:302] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 184c543cb05d49cc98ba1c56617be7a5; no voters:
I20240503 08:13:13.126802 9878 raft_consensus.cc:2781] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:13:13.126883 9878 raft_consensus.cc:482] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:13.126957 9878 raft_consensus.cc:3037] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:13.129487 9878 raft_consensus.cc:504] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:13.130339 9878 leader_election.cc:288] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 election: Requested vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
W20240503 08:13:13.130851 4902 leader_election.cc:334] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
I20240503 08:13:13.131163 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868bcace806f4d32977b21c876c2362b" candidate_uuid: "00b61625d8a7460892c118a2570afda5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "184c543cb05d49cc98ba1c56617be7a5"
I20240503 08:13:13.131428 2202 raft_consensus.cc:3037] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Advancing to term 2
W20240503 08:13:13.132184 4902 leader_election.cc:334] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
I20240503 08:13:13.133756 2202 raft_consensus.cc:2454] T 868bcace806f4d32977b21c876c2362b P 184c543cb05d49cc98ba1c56617be7a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b61625d8a7460892c118a2570afda5 in term 2.
I20240503 08:13:13.134265 4901 leader_election.cc:302] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00b61625d8a7460892c118a2570afda5, 184c543cb05d49cc98ba1c56617be7a5; no voters: b8f581390da449ccb9e299011296405e
I20240503 08:13:13.135360 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 1992
I20240503 08:13:13.137007 9878 raft_consensus.cc:2781] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:13.137214 9878 raft_consensus.cc:686] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [term 2 LEADER]: Becoming Leader. State: Replica: 00b61625d8a7460892c118a2570afda5, State: Running, Role: LEADER
I20240503 08:13:13.137490 9878 consensus_queue.cc:237] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:13.147692 1668 catalog_manager.cc:5484] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 reported cstate change: term changed from 1 to 2, leader changed from b8f581390da449ccb9e299011296405e (127.27.157.193) to 00b61625d8a7460892c118a2570afda5 (127.27.157.200). New cstate: current_term: 2 leader_uuid: "00b61625d8a7460892c118a2570afda5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 08:13:13.156306 9241 raft_consensus.cc:482] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:13.156528 9241 raft_consensus.cc:504] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:13:13.157279 9241 leader_election.cc:288] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:13:13.158080 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e36553b05f0491dbd70a6d7eb12d7f5" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc" is_pre_election: true
I20240503 08:13:13.158324 4703 raft_consensus.cc:2454] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 1.
I20240503 08:13:13.159355 4770 leader_election.cc:302] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters:
I20240503 08:13:13.159657 9241 raft_consensus.cc:2781] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240503 08:13:13.159673 4769 leader_election.cc:334] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
I20240503 08:13:13.159749 9241 raft_consensus.cc:482] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:13.159824 9241 raft_consensus.cc:3037] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:13.162488 9241 raft_consensus.cc:504] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:13:13.163230 9241 leader_election.cc:288] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 election: Requested vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
W20240503 08:13:13.167891 4769 leader_election.cc:334] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
I20240503 08:13:13.168244 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e36553b05f0491dbd70a6d7eb12d7f5" candidate_uuid: "f24648c92c384f1ba349eee801f985e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "31f41542452b4b1eb994ff4223d039fc"
I20240503 08:13:13.168476 4703 raft_consensus.cc:3037] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:13.170621 4703 raft_consensus.cc:2454] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24648c92c384f1ba349eee801f985e2 in term 2.
I20240503 08:13:13.171118 4770 leader_election.cc:302] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31f41542452b4b1eb994ff4223d039fc, f24648c92c384f1ba349eee801f985e2; no voters: b8f581390da449ccb9e299011296405e
I20240503 08:13:13.171308 9241 raft_consensus.cc:482] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:13.171738 10054 raft_consensus.cc:2781] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:13.171463 9241 raft_consensus.cc:504] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } }
I20240503 08:13:13.171885 10054 raft_consensus.cc:686] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [term 2 LEADER]: Becoming Leader. State: Replica: f24648c92c384f1ba349eee801f985e2, State: Running, Role: LEADER
I20240503 08:13:13.172127 10054 consensus_queue.cc:237] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } }
I20240503 08:13:13.178640 9241 leader_election.cc:288] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
I20240503 08:13:13.183727 1668 catalog_manager.cc:5484] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 reported cstate change: term changed from 1 to 2, leader changed from b8f581390da449ccb9e299011296405e (127.27.157.193) to f24648c92c384f1ba349eee801f985e2 (127.27.157.199). New cstate: current_term: 2 leader_uuid: "f24648c92c384f1ba349eee801f985e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
W20240503 08:13:13.187467 4769 leader_election.cc:334] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
W20240503 08:13:13.206208 4901 connection.cc:535] client connection to 127.27.157.195:43117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:13.206275 4768 connection.cc:535] client connection to 127.27.157.195:43117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:13.207821 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 2122
W20240503 08:13:13.208185 4901 consensus_peers.cc:485] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.208354 4901 consensus_peers.cc:485] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.209304 10083 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 08:13:13.207140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.157.195:43117 (local address 127.27.157.199:37219)
0503 08:13:13.208351 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0503 08:13:13.208442 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1078}
W20240503 08:13:13.209733 4768 consensus_peers.cc:485] T 4422e4f265f344039734893b759c2050 P f24648c92c384f1ba349eee801f985e2 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.209839 4768 consensus_peers.cc:485] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.209909 4768 leader_election.cc:334] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection reset by peer (error 104)
I20240503 08:13:13.210003 4768 leader_election.cc:302] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f24648c92c384f1ba349eee801f985e2; no voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e
I20240503 08:13:13.210328 10085 raft_consensus.cc:2726] T fc4d5ddaaa27463cba31eed19c4c1be0 P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:13:13.210338 4768 consensus_peers.cc:485] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.210568 4768 consensus_peers.cc:485] T 826e7d6e6d2947fbb2a178e8e2a32d56 P f24648c92c384f1ba349eee801f985e2 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.210645 4768 consensus_peers.cc:485] T f161c8da3b0e4b12b85ca0420a86dc07 P f24648c92c384f1ba349eee801f985e2 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.210700 4768 consensus_peers.cc:485] T 8ae7520f758c47a39cfa3aaf59ebc674 P f24648c92c384f1ba349eee801f985e2 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.217643 2243 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:13:13.246380 4636 consensus_peers.cc:485] T 0caddf5e39f8440baa8ad3656ce91a8e P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.257138 4768 consensus_peers.cc:485] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.263659 4767 consensus_peers.cc:485] T 3cda734f360c4d3eac88a6517028e49d P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:13.273332 4636 consensus_peers.cc:485] T e613a6cbac2c4ad9b71442baa71ba9a3 P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.273522 4636 consensus_peers.cc:485] T 7a792a309b3c43f482b4adb0fbcfadde P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.318269 4636 consensus_peers.cc:485] T 40476d1762154f6d9ce4855872492573 P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.385478 4636 consensus_peers.cc:485] T ebd7877c8faf409a8d042bef38f218aa P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.425923 4636 consensus_peers.cc:485] T eb4e33ffc60043d9a50f8381e85b927a P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.455823 4900 consensus_peers.cc:485] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:13.577095 4636 consensus_peers.cc:485] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.580317 4698 raft_consensus.cc:1261] T 3e36553b05f0491dbd70a6d7eb12d7f5 P 31f41542452b4b1eb994ff4223d039fc [term 2 FOLLOWER]: Refusing update from remote peer f24648c92c384f1ba349eee801f985e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 08:13:13.580847 9987 raft_consensus.cc:482] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:13.581027 9987 raft_consensus.cc:504] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:13.581568 10085 consensus_queue.cc:1035] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 08:13:13.581729 9987 leader_election.cc:288] T 5785af2b9ba34a3b95c5ffdb024b1c8f P 184c543cb05d49cc98ba1c56617be7a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), b8f581390da449ccb9e299011296405e (127.27.157.193:41181)
W20240503 08:13:13.596599 4636 consensus_peers.cc:485] T 8f3e931766d44f0aaf902cf0e950f83b P 31f41542452b4b1eb994ff4223d039fc -> Peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Couldn't send request to peer 43e2e01cdd314f9c872c4336ef74250a. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.597039 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 2122
W20240503 08:13:13.605561 4769 consensus_peers.cc:485] T 3e36553b05f0491dbd70a6d7eb12d7f5 P f24648c92c384f1ba349eee801f985e2 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.652813 4768 connection.cc:535] client connection to 127.27.157.196:42183 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:13.653462 4901 connection.cc:535] client connection to 127.27.157.196:42183 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:13.653734 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 4618
W20240503 08:13:13.653993 4636 connection.cc:535] client connection to 127.27.157.196:42183 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:13.654799 4901 consensus_peers.cc:485] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.654907 4901 consensus_peers.cc:485] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.655340 4768 consensus_peers.cc:485] T a03edb25e08e495c8dcb46adc9d8d2a4 P f24648c92c384f1ba349eee801f985e2 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.655452 4768 consensus_peers.cc:485] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.655519 4768 consensus_peers.cc:485] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.655825 4636 consensus_peers.cc:485] T fede923140fb45a5a4f93c0846b62595 P 31f41542452b4b1eb994ff4223d039fc -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.655845 4767 consensus_peers.cc:485] T f0c5c7ecb94e453f85d81b36974902ef P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:13.655889 4637 connection.cc:535] server connection from 127.27.157.196:40623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:13.655948 4636 consensus_peers.cc:485] T 01821d71873d4ad78d7b7f685161a229 P 31f41542452b4b1eb994ff4223d039fc -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.656018 4636 consensus_peers.cc:485] T 5d7781b8af904afbb8773a9e2f860388 P 31f41542452b4b1eb994ff4223d039fc -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.656095 4636 consensus_peers.cc:485] T 275fc5bbb1494306b1139b7f950ef80f P 31f41542452b4b1eb994ff4223d039fc -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:13.661437 4743 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:13:13.688447 4900 consensus_peers.cc:485] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:13.688594 4900 consensus_peers.cc:485] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:13:13.697983 10030 raft_consensus.cc:482] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:13.698180 10030 raft_consensus.cc:504] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:13.698786 10030 leader_election.cc:288] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:13.699254 10030 raft_consensus.cc:482] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:13.699433 10030 raft_consensus.cc:504] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
W20240503 08:13:13.700655 4902 leader_election.cc:334] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
I20240503 08:13:13.700235 10030 leader_election.cc:288] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
W20240503 08:13:13.700943 4900 leader_election.cc:334] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:13.701040 4900 leader_election.cc:302] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 5f95b05d111e4090bbc59dec394ed0ab, b8f581390da449ccb9e299011296405e
I20240503 08:13:13.701601 10030 raft_consensus.cc:2726] T 3219159f92774adbb7ecaa77ac267021 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:13:13.701795 4901 leader_election.cc:334] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111)
W20240503 08:13:13.701931 4902 leader_election.cc:334] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
I20240503 08:13:13.702039 4902 leader_election.cc:302] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 184c543cb05d49cc98ba1c56617be7a5, b8f581390da449ccb9e299011296405e
I20240503 08:13:13.702239 10030 raft_consensus.cc:2726] T 7d1e2530d5f2431baf0b2dd41081bb9d P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:13:13.709564 4901 consensus_peers.cc:485] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.721980 4900 consensus_peers.cc:485] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:13:13.761713 10085 raft_consensus.cc:482] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 315f3f0bcbf447a4b93b1aafcfbf0f5a)
I20240503 08:13:13.762008 10085 raft_consensus.cc:504] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { replace: true } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:13:13.762856 10085 leader_election.cc:288] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
W20240503 08:13:13.765362 4767 consensus_peers.cc:485] T 6427045c1be84157bfbc066642ff6505 P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:13.765713 4902 consensus_peers.cc:485] T 868bcace806f4d32977b21c876c2362b P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.766592 4768 consensus_peers.cc:485] T 820ecfef09a34327a08f7c3a893ccb05 P f24648c92c384f1ba349eee801f985e2 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.767190 4769 leader_election.cc:334] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111)
W20240503 08:13:13.767313 4768 leader_election.cc:334] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111)
I20240503 08:13:13.767411 4769 leader_election.cc:302] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f24648c92c384f1ba349eee801f985e2; no voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:13:13.767690 10085 raft_consensus.cc:2726] T a97625d919124c559b1e9a5cb785153f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:13:13.800307 4901 consensus_peers.cc:485] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.807862 4901 consensus_peers.cc:485] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.815232 4901 consensus_peers.cc:485] T 7f79a778fd444235a26fff1b713d6aac P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.815907 4901 consensus_peers.cc:485] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.867372 4901 consensus_peers.cc:485] T 6862b07fa36f4a458913803367c3f23e P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:13.938688 4900 consensus_peers.cc:485] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:13.961414 4900 consensus_peers.cc:485] T 09a268c672ba4580a2d93b5d15d12666 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 08:13:13.968698 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 4618
W20240503 08:13:13.970788 4767 consensus_peers.cc:485] T df0c416025c74075b6429d169fcdd58c P f24648c92c384f1ba349eee801f985e2 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:14.006352 4901 consensus_peers.cc:485] T b572ab5c63cf4317a754987c5e50d221 P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.015367 4900 consensus_peers.cc:485] T 296de1897a5e4969aa36b501080a35d9 P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:14.018467 4770 connection.cc:535] client connection to 127.27.157.198:46559 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 08:13:14.018872 4903 connection.cc:535] client connection to 127.27.157.198:46559 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:14.019140 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 4752
W20240503 08:13:14.020231 4903 consensus_peers.cc:485] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.020388 4903 consensus_peers.cc:485] T f3123fdbb5c945bd84f97fb4c35c5523 P 00b61625d8a7460892c118a2570afda5 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.020583 4770 consensus_peers.cc:485] T 54fe406b5f324d629243f7f939535620 P f24648c92c384f1ba349eee801f985e2 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.020718 4770 consensus_peers.cc:485] T 1d4f6c5975f8446a9f6ba8cae137bda0 P f24648c92c384f1ba349eee801f985e2 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.020812 4770 consensus_peers.cc:485] T f4a205bf277d4770836dbbf0ec28d884 P f24648c92c384f1ba349eee801f985e2 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.020892 4770 consensus_peers.cc:485] T 51f3edb7a72c4ba083429c150fa73363 P f24648c92c384f1ba349eee801f985e2 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.020973 4770 consensus_peers.cc:485] T 929d8d354b914502b5e677ccc84e2eda P f24648c92c384f1ba349eee801f985e2 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.021041 4770 consensus_peers.cc:485] T 98ad826dca7c4343afcae34bf1a40da8 P f24648c92c384f1ba349eee801f985e2 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.021131 4770 consensus_peers.cc:485] T 241f60a993f94046b870eae6ba173cc1 P f24648c92c384f1ba349eee801f985e2 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:14.026405 4874 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:13:14.063558 4902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111) [suppressed 167 similar messages]
W20240503 08:13:14.066443 4901 consensus_peers.cc:485] T dab60fefb4704ef19490fb3c76bf47ab P 00b61625d8a7460892c118a2570afda5 -> Peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Couldn't send request to peer 184c543cb05d49cc98ba1c56617be7a5. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.087596 4900 consensus_peers.cc:485] T c6994370a532472893cc934337f3cb6e P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:14.090883 4900 consensus_peers.cc:485] T 438146d5fa194156b929d7190b11a5de P 00b61625d8a7460892c118a2570afda5 -> Peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Couldn't send request to peer 5f95b05d111e4090bbc59dec394ed0ab. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 08:13:14.113133 4903 consensus_peers.cc:485] T 8737ef0936c641ccaf0b804b1d9c3e07 P 00b61625d8a7460892c118a2570afda5 -> Peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Couldn't send request to peer 31f41542452b4b1eb994ff4223d039fc. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.118290 4902 consensus_peers.cc:485] T 437037e814464cf5959153bcd3efc145 P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:14.148677 10108 raft_consensus.cc:482] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 315f3f0bcbf447a4b93b1aafcfbf0f5a)
I20240503 08:13:14.148785 10030 raft_consensus.cc:482] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 315f3f0bcbf447a4b93b1aafcfbf0f5a)
I20240503 08:13:14.149009 10108 raft_consensus.cc:504] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:14.149088 10030 raft_consensus.cc:504] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } attrs { promote: false } }
I20240503 08:13:14.149632 10109 raft_consensus.cc:482] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 315f3f0bcbf447a4b93b1aafcfbf0f5a)
I20240503 08:13:14.149907 10030 leader_election.cc:288] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
I20240503 08:13:14.150079 10030 raft_consensus.cc:482] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 315f3f0bcbf447a4b93b1aafcfbf0f5a)
I20240503 08:13:14.150206 10030 raft_consensus.cc:504] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:14.150765 10109 raft_consensus.cc:504] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:13:14.151917 10109 leader_election.cc:288] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
I20240503 08:13:14.150789 10030 leader_election.cc:288] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
W20240503 08:13:14.153260 4901 leader_election.cc:334] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111)
W20240503 08:13:14.153460 4901 leader_election.cc:334] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111)
I20240503 08:13:14.153642 10108 leader_election.cc:288] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
W20240503 08:13:14.153740 4902 leader_election.cc:334] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111)
I20240503 08:13:14.153856 4902 leader_election.cc:302] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:13:14.154114 10108 raft_consensus.cc:2726] T 6d69a5126dff43af8ba3acc931d17799 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:13:14.154150 4900 leader_election.cc:334] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:14.154263 4902 leader_election.cc:334] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111)
I20240503 08:13:14.154389 4902 leader_election.cc:302] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 43e2e01cdd314f9c872c4336ef74250a
I20240503 08:13:14.154625 10108 raft_consensus.cc:2726] T 9d11cd50b562408f9d3bc24a974bdba8 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:13:14.155354 4902 leader_election.cc:334] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111)
I20240503 08:13:14.155508 4902 leader_election.cc:302] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:14.155731 10108 raft_consensus.cc:2726] T 53197deeb8d7439c90b019a6a90857e0 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 08:13:14.156375 4902 leader_election.cc:334] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
W20240503 08:13:14.157227 4902 leader_election.cc:334] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111)
I20240503 08:13:14.157331 10108 raft_consensus.cc:482] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 315f3f0bcbf447a4b93b1aafcfbf0f5a)
I20240503 08:13:14.157357 4902 leader_election.cc:302] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, b8f581390da449ccb9e299011296405e
I20240503 08:13:14.157706 10030 raft_consensus.cc:2726] T fa111e798e3f46d2a5bc1eebfe28f8bf P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:14.157590 10108 raft_consensus.cc:504] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: false } }
I20240503 08:13:14.158221 10108 leader_election.cc:288] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
W20240503 08:13:14.158449 1656 catalog_manager.cc:4573] TS 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 51f3edb7a72c4ba083429c150fa73363: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
W20240503 08:13:14.159765 4902 leader_election.cc:334] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111)
W20240503 08:13:14.159965 4900 leader_election.cc:334] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465): Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111)
I20240503 08:13:14.160065 4900 leader_election.cc:302] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 315f3f0bcbf447a4b93b1aafcfbf0f5a, 5f95b05d111e4090bbc59dec394ed0ab
I20240503 08:13:14.160346 10108 raft_consensus.cc:2726] T f4900ffdbc624376b32f054efdb100b5 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:14.166811 10108 raft_consensus.cc:482] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 315f3f0bcbf447a4b93b1aafcfbf0f5a)
I20240503 08:13:14.167032 10108 raft_consensus.cc:504] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } } peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } }
I20240503 08:13:14.167734 10108 leader_election.cc:288] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975), 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183)
W20240503 08:13:14.169215 4901 leader_election.cc:334] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111)
W20240503 08:13:14.169433 4902 leader_election.cc:334] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975): Network error: Client connection negotiation failed: client connection to 127.27.157.194:41975: connect: Connection refused (error 111)
I20240503 08:13:14.169548 4902 leader_election.cc:302] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 184c543cb05d49cc98ba1c56617be7a5, 315f3f0bcbf447a4b93b1aafcfbf0f5a
I20240503 08:13:14.169787 10108 raft_consensus.cc:2726] T 3ebf60954ed44e29b5f8047ed511d2c7 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:14.327703 9241 raft_consensus.cc:482] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 43e2e01cdd314f9c872c4336ef74250a)
I20240503 08:13:14.327876 9874 raft_consensus.cc:482] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 43e2e01cdd314f9c872c4336ef74250a)
I20240503 08:13:14.328092 9874 raft_consensus.cc:504] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } attrs { promote: false } }
I20240503 08:13:14.329062 9874 leader_election.cc:288] T 84831077defe434c9f29eecfae7b44c8 P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:13:14.327908 9241 raft_consensus.cc:504] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } } peers { permanent_uuid: "315f3f0bcbf447a4b93b1aafcfbf0f5a" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 41975 } attrs { promote: false } }
I20240503 08:13:14.337901 9241 leader_election.cc:288] T 4d3aaa1025ca4b0e832535d1ca3d5c4f P f24648c92c384f1ba349eee801f985e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), 315f3f0bcbf447a4b93b1aafcfbf0f5a (127.27.157.194:41975)
I20240503 08:13:14.342305 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 4752
W20240503 08:13:14.396162 4903 connection.cc:535] client connection to 127.27.157.199:39455 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:14.397091 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 4884
W20240503 08:13:14.397235 4903 consensus_peers.cc:485] T 1cf9b2eef42e4fff8cdf5f9dd7b8b31e P 00b61625d8a7460892c118a2570afda5 -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.199:39455: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:13:14.401866 10030 raft_consensus.cc:482] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 184c543cb05d49cc98ba1c56617be7a5)
I20240503 08:13:14.402096 10030 raft_consensus.cc:504] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "184c543cb05d49cc98ba1c56617be7a5" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 42183 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } attrs { replace: true } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: NON_VOTER last_known_addr { host: "127.27.157.197" port: 42465 } attrs { promote: true } }
I20240503 08:13:14.402842 10030 leader_election.cc:288] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183), 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559)
W20240503 08:13:14.404984 4901 leader_election.cc:334] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 184c543cb05d49cc98ba1c56617be7a5 (127.27.157.196:42183): Network error: Client connection negotiation failed: client connection to 127.27.157.196:42183: connect: Connection refused (error 111)
W20240503 08:13:14.405273 4903 leader_election.cc:334] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111)
I20240503 08:13:14.405478 4903 leader_election.cc:302] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 184c543cb05d49cc98ba1c56617be7a5, 31f41542452b4b1eb994ff4223d039fc
I20240503 08:13:14.405866 10030 raft_consensus.cc:2726] T 0ba03a430b4b498f9ba66d733d68b13c P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:14.411371 5008 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:14.701710 10109 raft_consensus.cc:482] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b8f581390da449ccb9e299011296405e)
I20240503 08:13:14.702122 10109 raft_consensus.cc:504] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "5f95b05d111e4090bbc59dec394ed0ab" member_type: VOTER last_known_addr { host: "127.27.157.197" port: 42465 } }
I20240503 08:13:14.702942 10109 leader_election.cc:288] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 5f95b05d111e4090bbc59dec394ed0ab (127.27.157.197:42465)
W20240503 08:13:14.716315 4902 consensus_peers.cc:485] T 733bed28775e4779b74e92d8a395234f P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:14.716526 4902 consensus_peers.cc:485] T 34b7fd8dc7434b8ea14b9deb0023b4cb P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 08:13:14.716648 4902 leader_election.cc:334] T 09ae3cc0ce534128b49f2da744d23144 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
W20240503 08:13:14.716809 4902 consensus_peers.cc:485] T 9a0bffb0aa304bd184b8ad26bc94fd18 P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:14.717348 10124 raft_consensus.cc:482] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 43e2e01cdd314f9c872c4336ef74250a)
I20240503 08:13:14.717576 10124 raft_consensus.cc:504] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } } peers { permanent_uuid: "b8f581390da449ccb9e299011296405e" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 41181 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } }
I20240503 08:13:14.718155 10124 leader_election.cc:288] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8f581390da449ccb9e299011296405e (127.27.157.193:41181), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
I20240503 08:13:14.718675 10122 raft_consensus.cc:482] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 43e2e01cdd314f9c872c4336ef74250a)
I20240503 08:13:14.718830 10122 raft_consensus.cc:504] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "f24648c92c384f1ba349eee801f985e2" member_type: VOTER last_known_addr { host: "127.27.157.199" port: 39455 } attrs { promote: false } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
W20240503 08:13:14.719488 4901 leader_election.cc:334] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117): Network error: Client connection negotiation failed: client connection to 127.27.157.195:43117: connect: Connection refused (error 111)
W20240503 08:13:14.720387 4902 consensus_peers.cc:485] T 415a75c852814ea89b65d294d682c571 P 00b61625d8a7460892c118a2570afda5 -> Peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Couldn't send request to peer b8f581390da449ccb9e299011296405e. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:13:14.720980 10122 leader_election.cc:288] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117), f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455)
W20240503 08:13:14.723086 4903 leader_election.cc:334] T 044d8e5a69444865980ffc06a6be9080 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Network error: Client connection negotiation failed: client connection to 127.27.157.199:39455: connect: Connection refused (error 111)
I20240503 08:13:14.723421 10122 raft_consensus.cc:482] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31f41542452b4b1eb994ff4223d039fc)
I20240503 08:13:14.723610 10122 raft_consensus.cc:504] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "31f41542452b4b1eb994ff4223d039fc" member_type: VOTER last_known_addr { host: "127.27.157.198" port: 46559 } } peers { permanent_uuid: "43e2e01cdd314f9c872c4336ef74250a" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 43117 } } peers { permanent_uuid: "00b61625d8a7460892c118a2570afda5" member_type: VOTER last_known_addr { host: "127.27.157.200" port: 45283 } attrs { promote: false } }
I20240503 08:13:14.724355 10122 leader_election.cc:288] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559), 43e2e01cdd314f9c872c4336ef74250a (127.27.157.195:43117)
W20240503 08:13:14.725601 4903 consensus_peers.cc:485] T a76b5650ad0b400da5db935edd1df6b9 P 00b61625d8a7460892c118a2570afda5 -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.199:39455: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.731655 4903 consensus_peers.cc:485] T 4fba9f969e764737973456dbdb650b1d P 00b61625d8a7460892c118a2570afda5 -> Peer f24648c92c384f1ba349eee801f985e2 (127.27.157.199:39455): Couldn't send request to peer f24648c92c384f1ba349eee801f985e2. Status: Network error: Client connection negotiation failed: client connection to 127.27.157.199:39455: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:13:14.731889 4903 leader_election.cc:334] T e613a6cbac2c4ad9b71442baa71ba9a3 P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 31f41542452b4b1eb994ff4223d039fc (127.27.157.198:46559): Network error: Client connection negotiation failed: client connection to 127.27.157.198:46559: connect: Connection refused (error 111)
W20240503 08:13:14.732496 4902 leader_election.cc:334] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8f581390da449ccb9e299011296405e (127.27.157.193:41181): Network error: Client connection negotiation failed: client connection to 127.27.157.193:41181: connect: Connection refused (error 111)
I20240503 08:13:14.732854 4902 leader_election.cc:302] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00b61625d8a7460892c118a2570afda5; no voters: 43e2e01cdd314f9c872c4336ef74250a, b8f581390da449ccb9e299011296405e
I20240503 08:13:14.733249 10124 raft_consensus.cc:2726] T 938428b201e842448232a1d77061afab P 00b61625d8a7460892c118a2570afda5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 08:13:14.733667 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 4884
W20240503 08:13:14.780344 1656 connection.cc:535] server connection from 127.27.157.200:43103 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 08:13:14.780814 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 1641
I20240503 08:13:14.787122 1701 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 08:13:15.073632 1656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.157.197:42465: connect: Connection refused (error 111) [suppressed 16 similar messages]
I20240503 08:13:15.073756 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 1641
2024-05-03T08:13:15Z chronyd exiting
I20240503 08:13:15.142262 28279 test_util.cc:175] -----------------------------------------------
I20240503 08:13:15.142372 28279 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723865426541-28279-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1, where GetParam() = 4-byte object <01-00 00-00> (82483 ms)
[----------] 1 test from DDLDuringRebalancingTest (82483 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/1
I20240503 08:13:15.144800 28279 test_util.cc:268] Using random seed: 1251889717
I20240503 08:13:15.146865 28279 ts_itest-base.cc:112] Starting cluster with:
I20240503 08:13:15.146976 28279 ts_itest-base.cc:113] --------------
I20240503 08:13:15.147034 28279 ts_itest-base.cc:114] 8 tablet servers
I20240503 08:13:15.147101 28279 ts_itest-base.cc:115] 3 replicas per TS
I20240503 08:13:15.147152 28279 ts_itest-base.cc:116] --------------
2024-05-03T08:13:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T08:13:15Z Disabled control of system clock
I20240503 08:13:15.181651 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.157.254:35531
--webserver_interface=127.27.157.254
--webserver_port=0
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.157.254:35531
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 08:13:15.555173 10137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:15.555611 10137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:15.555755 10137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:15.563813 10137 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:15.563993 10137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 08:13:15.564085 10137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:15.564183 10137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 08:13:15.564262 10137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 08:13:15.574337 10137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.157.254:35531
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.157.254:35531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.157.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:15.575841 10137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:15.577802 10137 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:15.590332 10142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:15.590974 10143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:15.591575 10145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:15.592679 10137 server_base.cc:1034] running on GCE node
I20240503 08:13:15.593660 10137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:15.595315 10137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:15.596611 10137 hybrid_clock.cc:648] HybridClock initialized: now 1714723995596537 us; error 71 us; skew 500 ppm
I20240503 08:13:15.597062 10137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:15.599962 10137 webserver.cc:460] Webserver started at http://127.27.157.254:43409/ using document root <none> and password file <none>
I20240503 08:13:15.600898 10137 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:15.601032 10137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:15.601439 10137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:15.604562 10137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9865283f70e84701b663342968213c17"
format_stamp: "Formatted at 2024-05-03 08:13:15 on dist-test-slave-dj8w"
I20240503 08:13:15.605552 10137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9865283f70e84701b663342968213c17"
format_stamp: "Formatted at 2024-05-03 08:13:15 on dist-test-slave-dj8w"
I20240503 08:13:15.611858 10137 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20240503 08:13:15.616166 10151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:15.617902 10137 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:13:15.618175 10137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9865283f70e84701b663342968213c17"
format_stamp: "Formatted at 2024-05-03 08:13:15 on dist-test-slave-dj8w"
I20240503 08:13:15.618460 10137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:15.648491 10137 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:15.649468 10137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:15.649785 10137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:15.670301 10137 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.254:35531
I20240503 08:13:15.670307 10202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.254:35531 every 8 connection(s)
I20240503 08:13:15.671883 10137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 08:13:15.675944 10203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:15.676589 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 10137
I20240503 08:13:15.676822 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 08:13:15.690459 10203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17: Bootstrap starting.
I20240503 08:13:15.694175 10203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:15.695487 10203 log.cc:824] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:15.698590 10203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17: No bootstrap required, opened a new log
I20240503 08:13:15.705763 10203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9865283f70e84701b663342968213c17" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 35531 } }
I20240503 08:13:15.706203 10203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:15.706324 10203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9865283f70e84701b663342968213c17, State: Initialized, Role: FOLLOWER
I20240503 08:13:15.706997 10203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9865283f70e84701b663342968213c17" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 35531 } }
I20240503 08:13:15.707228 10203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:13:15.707377 10203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:13:15.707561 10203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:15.710065 10203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9865283f70e84701b663342968213c17" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 35531 } }
I20240503 08:13:15.710743 10203 leader_election.cc:302] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9865283f70e84701b663342968213c17; no voters:
I20240503 08:13:15.711395 10203 leader_election.cc:288] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:13:15.711557 10208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:15.712237 10208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [term 1 LEADER]: Becoming Leader. State: Replica: 9865283f70e84701b663342968213c17, State: Running, Role: LEADER
I20240503 08:13:15.712872 10208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9865283f70e84701b663342968213c17" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 35531 } }
I20240503 08:13:15.713523 10203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:13:15.716200 10209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9865283f70e84701b663342968213c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9865283f70e84701b663342968213c17" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 35531 } } }
I20240503 08:13:15.716701 10209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:15.716521 10210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9865283f70e84701b663342968213c17. Latest consensus state: current_term: 1 leader_uuid: "9865283f70e84701b663342968213c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9865283f70e84701b663342968213c17" member_type: VOTER last_known_addr { host: "127.27.157.254" port: 35531 } } }
I20240503 08:13:15.716823 10210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17 [sys.catalog]: This master's current role is: LEADER
I20240503 08:13:15.719264 10215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:13:15.725245 10215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:13:15.735986 10215 catalog_manager.cc:1347] Generated new cluster ID: b46ec12ca577484486fcf53ea2d753dd
I20240503 08:13:15.736112 10215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:13:15.747303 10215 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:13:15.748678 10215 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:13:15.760919 10215 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9865283f70e84701b663342968213c17: Generated new TSK 0
I20240503 08:13:15.761968 10215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:13:15.774766 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.193:0
--local_ip_for_outbound_sockets=127.27.157.193
--webserver_interface=127.27.157.193
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:13:16.142938 10227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:16.143287 10227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:16.143468 10227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:16.151486 10227 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:16.151634 10227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:16.151790 10227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.193
I20240503 08:13:16.161468 10227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.157.193
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.193
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:16.162927 10227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:16.164921 10227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:16.178400 10233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:16.179011 10235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:16.179626 10232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:16.180593 10227 server_base.cc:1034] running on GCE node
I20240503 08:13:16.181298 10227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:16.182415 10227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:16.183676 10227 hybrid_clock.cc:648] HybridClock initialized: now 1714723996183604 us; error 64 us; skew 500 ppm
I20240503 08:13:16.184101 10227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:16.187036 10227 webserver.cc:460] Webserver started at http://127.27.157.193:34361/ using document root <none> and password file <none>
I20240503 08:13:16.187974 10227 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:16.188119 10227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:16.188482 10227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:16.191457 10227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
format_stamp: "Formatted at 2024-05-03 08:13:16 on dist-test-slave-dj8w"
I20240503 08:13:16.192318 10227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
format_stamp: "Formatted at 2024-05-03 08:13:16 on dist-test-slave-dj8w"
I20240503 08:13:16.198377 10227 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240503 08:13:16.202598 10241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:16.204337 10227 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:16.204571 10227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
format_stamp: "Formatted at 2024-05-03 08:13:16 on dist-test-slave-dj8w"
I20240503 08:13:16.204815 10227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:16.232540 10227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:16.233506 10227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:16.233808 10227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:16.234887 10227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:16.236678 10227 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:16.236795 10227 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:16.236938 10227 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:16.237018 10227 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:16.272760 10227 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.193:38359
I20240503 08:13:16.272806 10353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.193:38359 every 8 connection(s)
I20240503 08:13:16.274242 10227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 08:13:16.280019 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 10227
I20240503 08:13:16.280225 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 08:13:16.283633 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.194:0
--local_ip_for_outbound_sockets=127.27.157.194
--webserver_interface=127.27.157.194
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:16.295279 10354 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:16.295720 10354 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:16.296916 10354 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:16.299780 10168 ts_manager.cc:194] Registered new tserver with Master: d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:16.302379 10168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.193:47507
W20240503 08:13:16.648947 10358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:16.649425 10358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:16.649680 10358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:16.662664 10358 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:16.662915 10358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:16.663142 10358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.194
I20240503 08:13:16.673135 10358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.157.194
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.194
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:16.674584 10358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:16.676594 10358 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:16.690701 10363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:16.691118 10366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:16.691803 10364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:16.692572 10358 server_base.cc:1034] running on GCE node
I20240503 08:13:16.693272 10358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:16.694408 10358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:16.695672 10358 hybrid_clock.cc:648] HybridClock initialized: now 1714723996695621 us; error 42 us; skew 500 ppm
I20240503 08:13:16.696043 10358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:16.698740 10358 webserver.cc:460] Webserver started at http://127.27.157.194:46369/ using document root <none> and password file <none>
I20240503 08:13:16.699618 10358 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:16.699741 10358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:16.700073 10358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:16.703028 10358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2b3a6ff465524577a400de6af7e70f71"
format_stamp: "Formatted at 2024-05-03 08:13:16 on dist-test-slave-dj8w"
I20240503 08:13:16.703999 10358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2b3a6ff465524577a400de6af7e70f71"
format_stamp: "Formatted at 2024-05-03 08:13:16 on dist-test-slave-dj8w"
I20240503 08:13:16.710773 10358 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20240503 08:13:16.715123 10372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:16.716856 10358 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:16.717087 10358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2b3a6ff465524577a400de6af7e70f71"
format_stamp: "Formatted at 2024-05-03 08:13:16 on dist-test-slave-dj8w"
I20240503 08:13:16.717327 10358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:16.758658 10358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:16.759654 10358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:16.759946 10358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:16.761025 10358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:16.762778 10358 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:16.762876 10358 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:16.762959 10358 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:16.763016 10358 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:16.797155 10358 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.194:39249
I20240503 08:13:16.797216 10484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.194:39249 every 8 connection(s)
I20240503 08:13:16.798595 10358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 08:13:16.800532 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 10358
I20240503 08:13:16.800740 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 08:13:16.804855 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.195:0
--local_ip_for_outbound_sockets=127.27.157.195
--webserver_interface=127.27.157.195
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:16.815783 10485 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:16.816114 10485 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:16.817057 10485 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:16.818902 10168 ts_manager.cc:194] Registered new tserver with Master: 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:16.820255 10168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.194:56777
W20240503 08:13:17.166471 10489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:17.166814 10489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:17.166908 10489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:17.174762 10489 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:17.174908 10489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:17.175086 10489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.195
I20240503 08:13:17.184899 10489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.157.195
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.195
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:17.186345 10489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:17.188361 10489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:17.202190 10497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:17.202739 10495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:17.203518 10494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:17.204155 10489 server_base.cc:1034] running on GCE node
I20240503 08:13:17.204893 10489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:17.206017 10489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:17.207270 10489 hybrid_clock.cc:648] HybridClock initialized: now 1714723997207210 us; error 58 us; skew 500 ppm
I20240503 08:13:17.207791 10489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:17.210618 10489 webserver.cc:460] Webserver started at http://127.27.157.195:39809/ using document root <none> and password file <none>
I20240503 08:13:17.211515 10489 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:17.211644 10489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:17.211997 10489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:17.215147 10489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "22588cae35464393b2d6adb27e3114b2"
format_stamp: "Formatted at 2024-05-03 08:13:17 on dist-test-slave-dj8w"
I20240503 08:13:17.216070 10489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "22588cae35464393b2d6adb27e3114b2"
format_stamp: "Formatted at 2024-05-03 08:13:17 on dist-test-slave-dj8w"
I20240503 08:13:17.222127 10489 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240503 08:13:17.226361 10503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:17.228147 10489 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:17.228356 10489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "22588cae35464393b2d6adb27e3114b2"
format_stamp: "Formatted at 2024-05-03 08:13:17 on dist-test-slave-dj8w"
I20240503 08:13:17.228605 10489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:17.259303 10489 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:17.260331 10489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:17.260648 10489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:17.261777 10489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:17.263548 10489 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:17.263649 10489 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:17.263770 10489 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:17.263861 10489 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:17.300052 10489 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.195:44929
I20240503 08:13:17.300134 10615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.195:44929 every 8 connection(s)
I20240503 08:13:17.301522 10489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 08:13:17.302088 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 10489
I20240503 08:13:17.302307 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 08:13:17.306335 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.196:0
--local_ip_for_outbound_sockets=127.27.157.196
--webserver_interface=127.27.157.196
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:17.307178 10354 heartbeater.cc:503] Master 127.27.157.254:35531 was elected leader, sending a full tablet report...
I20240503 08:13:17.318363 10616 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:17.318795 10616 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:17.320106 10616 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:17.322240 10168 ts_manager.cc:194] Registered new tserver with Master: 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:17.323259 10168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.195:45903
W20240503 08:13:17.671169 10619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:17.671543 10619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:17.671691 10619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:17.679715 10619 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:17.679901 10619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:17.680075 10619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.196
I20240503 08:13:17.689726 10619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.157.196
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.196
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:17.691186 10619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:17.693168 10619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:17.706514 10624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:17.707118 10627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:17.707835 10625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:17.708667 10619 server_base.cc:1034] running on GCE node
I20240503 08:13:17.709431 10619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:17.710615 10619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:17.711845 10619 hybrid_clock.cc:648] HybridClock initialized: now 1714723997711788 us; error 53 us; skew 500 ppm
I20240503 08:13:17.712167 10619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:17.714969 10619 webserver.cc:460] Webserver started at http://127.27.157.196:46357/ using document root <none> and password file <none>
I20240503 08:13:17.715847 10619 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:17.715963 10619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:17.716287 10619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:17.719053 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "54534138f6814b809a25c03a46e5f931"
format_stamp: "Formatted at 2024-05-03 08:13:17 on dist-test-slave-dj8w"
I20240503 08:13:17.720165 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "54534138f6814b809a25c03a46e5f931"
format_stamp: "Formatted at 2024-05-03 08:13:17 on dist-test-slave-dj8w"
I20240503 08:13:17.726668 10619 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240503 08:13:17.730711 10633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:17.732568 10619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:17.732808 10619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "54534138f6814b809a25c03a46e5f931"
format_stamp: "Formatted at 2024-05-03 08:13:17 on dist-test-slave-dj8w"
I20240503 08:13:17.733064 10619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:17.756685 10619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:17.757598 10619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:17.757867 10619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:17.758850 10619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:17.760541 10619 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:17.760641 10619 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:17.760740 10619 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:17.760797 10619 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:17.795717 10619 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.196:36257
I20240503 08:13:17.795794 10745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.196:36257 every 8 connection(s)
I20240503 08:13:17.797188 10619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 08:13:17.805083 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 10619
I20240503 08:13:17.805308 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 08:13:17.809082 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.197:0
--local_ip_for_outbound_sockets=127.27.157.197
--webserver_interface=127.27.157.197
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:17.813320 10746 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:17.813716 10746 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:17.814734 10746 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:17.816424 10168 ts_manager.cc:194] Registered new tserver with Master: 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:17.817297 10168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.196:52055
I20240503 08:13:17.823490 10485 heartbeater.cc:503] Master 127.27.157.254:35531 was elected leader, sending a full tablet report...
W20240503 08:13:18.175698 10749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:18.176028 10749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:18.176142 10749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:18.184192 10749 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:18.184340 10749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:18.184497 10749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.197
I20240503 08:13:18.194221 10749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.157.197
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.197
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:18.195789 10749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:18.197722 10749 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:18.211516 10754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:18.212184 10755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:18.212769 10757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:18.213652 10749 server_base.cc:1034] running on GCE node
I20240503 08:13:18.214356 10749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:18.215559 10749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:18.216792 10749 hybrid_clock.cc:648] HybridClock initialized: now 1714723998216736 us; error 47 us; skew 500 ppm
I20240503 08:13:18.217155 10749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:18.219909 10749 webserver.cc:460] Webserver started at http://127.27.157.197:42783/ using document root <none> and password file <none>
I20240503 08:13:18.220757 10749 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:18.220880 10749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:18.221228 10749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:18.224522 10749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b327eb1caa0d40e58fc9dbb2409fa85e"
format_stamp: "Formatted at 2024-05-03 08:13:18 on dist-test-slave-dj8w"
I20240503 08:13:18.225342 10749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b327eb1caa0d40e58fc9dbb2409fa85e"
format_stamp: "Formatted at 2024-05-03 08:13:18 on dist-test-slave-dj8w"
I20240503 08:13:18.231444 10749 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 08:13:18.235999 10763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:18.237740 10749 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:13:18.237964 10749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b327eb1caa0d40e58fc9dbb2409fa85e"
format_stamp: "Formatted at 2024-05-03 08:13:18 on dist-test-slave-dj8w"
I20240503 08:13:18.238214 10749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:18.268550 10749 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:18.269497 10749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:18.269784 10749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:18.270830 10749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:18.272598 10749 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:18.272711 10749 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:18.272828 10749 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:18.272909 10749 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:18.310328 10749 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.197:43391
I20240503 08:13:18.310387 10875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.197:43391 every 8 connection(s)
I20240503 08:13:18.311959 10749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:13:18.316027 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 10749
I20240503 08:13:18.316243 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 08:13:18.319839 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.198:0
--local_ip_for_outbound_sockets=127.27.157.198
--webserver_interface=127.27.157.198
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:18.326311 10616 heartbeater.cc:503] Master 127.27.157.254:35531 was elected leader, sending a full tablet report...
I20240503 08:13:18.327502 10876 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:18.327904 10876 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:18.329097 10876 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:18.331002 10168 ts_manager.cc:194] Registered new tserver with Master: b327eb1caa0d40e58fc9dbb2409fa85e (127.27.157.197:43391)
I20240503 08:13:18.331951 10168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.197:38329
W20240503 08:13:18.683254 10880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:18.683661 10880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:18.683816 10880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:18.691790 10880 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:18.691968 10880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:18.692150 10880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.198
I20240503 08:13:18.701793 10880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.157.198
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.198
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:18.703313 10880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:18.705322 10880 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:18.719341 10885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:18.720057 10886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:18.720758 10888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:18.721475 10880 server_base.cc:1034] running on GCE node
I20240503 08:13:18.722198 10880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:18.723392 10880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:18.724617 10880 hybrid_clock.cc:648] HybridClock initialized: now 1714723998724562 us; error 59 us; skew 500 ppm
I20240503 08:13:18.725037 10880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:18.727680 10880 webserver.cc:460] Webserver started at http://127.27.157.198:33089/ using document root <none> and password file <none>
I20240503 08:13:18.728490 10880 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:18.728610 10880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:18.728951 10880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:18.731835 10880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "22887faab4ec4e3caa006fee5b91099b"
format_stamp: "Formatted at 2024-05-03 08:13:18 on dist-test-slave-dj8w"
I20240503 08:13:18.732719 10880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "22887faab4ec4e3caa006fee5b91099b"
format_stamp: "Formatted at 2024-05-03 08:13:18 on dist-test-slave-dj8w"
I20240503 08:13:18.738745 10880 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240503 08:13:18.742648 10894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:18.744504 10880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:18.744714 10880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "22887faab4ec4e3caa006fee5b91099b"
format_stamp: "Formatted at 2024-05-03 08:13:18 on dist-test-slave-dj8w"
I20240503 08:13:18.744925 10880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:18.767063 10880 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:18.768046 10880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:18.768338 10880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:18.769353 10880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:18.771005 10880 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:18.771104 10880 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:18.771251 10880 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:18.771351 10880 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:18.806787 10880 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.198:40735
I20240503 08:13:18.806871 11006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.198:40735 every 8 connection(s)
I20240503 08:13:18.808595 10880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:13:18.818567 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 10880
I20240503 08:13:18.818782 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 08:13:18.820425 10746 heartbeater.cc:503] Master 127.27.157.254:35531 was elected leader, sending a full tablet report...
I20240503 08:13:18.822597 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.199:0
--local_ip_for_outbound_sockets=127.27.157.199
--webserver_interface=127.27.157.199
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:18.823894 11007 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:18.824306 11007 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:18.825511 11007 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:18.827370 10167 ts_manager.cc:194] Registered new tserver with Master: 22887faab4ec4e3caa006fee5b91099b (127.27.157.198:40735)
I20240503 08:13:18.828676 10167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.198:32951
W20240503 08:13:19.186439 11010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:19.186796 11010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:19.186954 11010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:19.194938 11010 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:19.195094 11010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:19.195273 11010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.199
I20240503 08:13:19.205219 11010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.157.199
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.199
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:19.206674 11010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:19.208626 11010 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:19.222834 11015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:19.223212 11016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:19.224002 11018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:19.229538 11010 server_base.cc:1034] running on GCE node
I20240503 08:13:19.230345 11010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:19.231652 11010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:19.232990 11010 hybrid_clock.cc:648] HybridClock initialized: now 1714723999232932 us; error 68 us; skew 500 ppm
I20240503 08:13:19.233472 11010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:19.236410 11010 webserver.cc:460] Webserver started at http://127.27.157.199:42845/ using document root <none> and password file <none>
I20240503 08:13:19.237258 11010 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:19.237396 11010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:19.237751 11010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:19.240754 11010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "e14ef04b4a534fc1aa67ebfe5bdfceeb"
format_stamp: "Formatted at 2024-05-03 08:13:19 on dist-test-slave-dj8w"
I20240503 08:13:19.241616 11010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "e14ef04b4a534fc1aa67ebfe5bdfceeb"
format_stamp: "Formatted at 2024-05-03 08:13:19 on dist-test-slave-dj8w"
I20240503 08:13:19.248167 11010 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 08:13:19.252472 11024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:19.254189 11010 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:13:19.254417 11010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e14ef04b4a534fc1aa67ebfe5bdfceeb"
format_stamp: "Formatted at 2024-05-03 08:13:19 on dist-test-slave-dj8w"
I20240503 08:13:19.254658 11010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:19.277743 11010 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:19.278755 11010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:19.279069 11010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:19.280182 11010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:19.281816 11010 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:19.281920 11010 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:19.282058 11010 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:19.282142 11010 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:19.321950 11010 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.199:35937
I20240503 08:13:19.321993 11136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.199:35937 every 8 connection(s)
I20240503 08:13:19.323693 11010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:13:19.329777 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 11010
I20240503 08:13:19.329983 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 08:13:19.333848 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.200:0
--local_ip_for_outbound_sockets=127.27.157.200
--webserver_interface=127.27.157.200
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:19.335193 10876 heartbeater.cc:503] Master 127.27.157.254:35531 was elected leader, sending a full tablet report...
I20240503 08:13:19.340186 11137 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:19.340605 11137 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:19.341578 11137 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:19.343477 10167 ts_manager.cc:194] Registered new tserver with Master: e14ef04b4a534fc1aa67ebfe5bdfceeb (127.27.157.199:35937)
I20240503 08:13:19.344763 10167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.199:60937
W20240503 08:13:19.697024 11141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:19.697444 11141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:19.697600 11141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:19.705734 11141 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:19.705909 11141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:19.706084 11141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.200
I20240503 08:13:19.716076 11141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.157.200
--webserver_port=0
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.200
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:19.717479 11141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:19.719460 11141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:19.734402 11146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:19.734768 11147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:19.735648 11149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:19.736493 11141 server_base.cc:1034] running on GCE node
I20240503 08:13:19.737214 11141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:19.738451 11141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:19.739732 11141 hybrid_clock.cc:648] HybridClock initialized: now 1714723999739675 us; error 48 us; skew 500 ppm
I20240503 08:13:19.740128 11141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:19.749768 11141 webserver.cc:460] Webserver started at http://127.27.157.200:37353/ using document root <none> and password file <none>
I20240503 08:13:19.750746 11141 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:19.750896 11141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:19.751375 11141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:13:19.755864 11141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "4a4e71bb1f1c41afbaf4497af3ddd640"
format_stamp: "Formatted at 2024-05-03 08:13:19 on dist-test-slave-dj8w"
I20240503 08:13:19.757547 11141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "4a4e71bb1f1c41afbaf4497af3ddd640"
format_stamp: "Formatted at 2024-05-03 08:13:19 on dist-test-slave-dj8w"
I20240503 08:13:19.766738 11141 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.004s
I20240503 08:13:19.772363 11155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:19.774219 11141 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:13:19.774415 11141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4a4e71bb1f1c41afbaf4497af3ddd640"
format_stamp: "Formatted at 2024-05-03 08:13:19 on dist-test-slave-dj8w"
I20240503 08:13:19.774721 11141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:19.806860 11141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:19.807873 11141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:19.808172 11141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:19.809329 11141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:19.811218 11141 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:19.811847 11141 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:19.812000 11141 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:19.812089 11141 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:19.831662 11007 heartbeater.cc:503] Master 127.27.157.254:35531 was elected leader, sending a full tablet report...
I20240503 08:13:19.847532 11267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.200:45347 every 8 connection(s)
I20240503 08:13:19.847813 11141 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.200:45347
I20240503 08:13:19.849303 11141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:13:19.853008 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 11141
I20240503 08:13:19.853217 28279 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 08:13:19.872934 11268 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:19.873309 11268 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:19.874406 11268 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:19.876274 10167 ts_manager.cc:194] Registered new tserver with Master: 4a4e71bb1f1c41afbaf4497af3ddd640 (127.27.157.200:45347)
I20240503 08:13:19.877120 10167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.200:34873
I20240503 08:13:19.885385 28279 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:13:19.904830 28279 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 08:13:19.904978 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 10749
I20240503 08:13:19.914258 10871 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:20.000255 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 10749
I20240503 08:13:20.017333 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 10880
I20240503 08:13:20.027666 11002 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:20.114120 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 10880
I20240503 08:13:20.130791 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 11010
I20240503 08:13:20.140033 11132 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:20.224208 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 11010
I20240503 08:13:20.240775 28279 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpimNqO/build/asan/bin/kudu pid 11141
I20240503 08:13:20.250838 11263 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 08:13:20.336618 28279 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpimNqO/build/asan/bin/kudu with pid 11141
I20240503 08:13:24.105928 10167 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44620:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:24.157716 10419 tablet_service.cc:1467] Processing CreateTablet for tablet 57310c72619e44c99fa7f42673648658 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de68fe80e1c49ec953529b4fc4ff5ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:24.159194 10420 tablet_service.cc:1467] Processing CreateTablet for tablet 3715b584822e4cd3bee05119936125d0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de68fe80e1c49ec953529b4fc4ff5ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:24.159533 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57310c72619e44c99fa7f42673648658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:24.160071 10420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3715b584822e4cd3bee05119936125d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:24.159971 10289 tablet_service.cc:1467] Processing CreateTablet for tablet 3715b584822e4cd3bee05119936125d0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de68fe80e1c49ec953529b4fc4ff5ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:24.161528 10289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3715b584822e4cd3bee05119936125d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:24.162187 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 57310c72619e44c99fa7f42673648658 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de68fe80e1c49ec953529b4fc4ff5ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:24.162844 10550 tablet_service.cc:1467] Processing CreateTablet for tablet e8127b76db73402d8100faad68271644 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de68fe80e1c49ec953529b4fc4ff5ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:24.163854 10550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8127b76db73402d8100faad68271644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:24.166129 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57310c72619e44c99fa7f42673648658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:24.162151 10287 tablet_service.cc:1467] Processing CreateTablet for tablet e8127b76db73402d8100faad68271644 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de68fe80e1c49ec953529b4fc4ff5ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:24.167866 10288 tablet_service.cc:1467] Processing CreateTablet for tablet 57310c72619e44c99fa7f42673648658 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de68fe80e1c49ec953529b4fc4ff5ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:24.174429 10681 tablet_service.cc:1467] Processing CreateTablet for tablet 3715b584822e4cd3bee05119936125d0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de68fe80e1c49ec953529b4fc4ff5ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:24.175877 10680 tablet_service.cc:1467] Processing CreateTablet for tablet e8127b76db73402d8100faad68271644 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2de68fe80e1c49ec953529b4fc4ff5ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:24.177170 10681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3715b584822e4cd3bee05119936125d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:24.177839 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8127b76db73402d8100faad68271644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:24.178409 10288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57310c72619e44c99fa7f42673648658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:24.183606 10287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8127b76db73402d8100faad68271644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:24.203830 11300 tablet_bootstrap.cc:492] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:24.205729 11299 tablet_bootstrap.cc:492] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:24.208530 11300 tablet_bootstrap.cc:654] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:24.209904 11298 tablet_bootstrap.cc:492] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:24.210268 11299 tablet_bootstrap.cc:654] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:24.211283 11299 log.cc:824] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:24.210973 11300 log.cc:824] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:24.213989 11301 tablet_bootstrap.cc:492] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:24.216713 11300 tablet_bootstrap.cc:492] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:24.217110 11300 ts_tablet_manager.cc:1367] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240503 08:13:24.217736 11301 tablet_bootstrap.cc:654] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:24.219184 11301 log.cc:824] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:24.219834 11298 tablet_bootstrap.cc:654] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:24.221652 11298 log.cc:824] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71: Log is configured to *not* fsync() on all Append() calls
I20240503 08:13:24.225950 11299 tablet_bootstrap.cc:492] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:24.226374 11299 ts_tablet_manager.cc:1367] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.011s
I20240503 08:13:24.229336 11298 tablet_bootstrap.cc:492] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:24.229336 11301 tablet_bootstrap.cc:492] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:24.229676 11298 ts_tablet_manager.cc:1367] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20240503 08:13:24.229676 11301 ts_tablet_manager.cc:1367] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20240503 08:13:24.229876 11300 raft_consensus.cc:348] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:24.230417 11300 raft_consensus.cc:374] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:24.230562 11300 raft_consensus.cc:729] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:24.231457 11300 consensus_queue.cc:260] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:24.238418 11299 raft_consensus.cc:348] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:24.239267 11299 raft_consensus.cc:374] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:24.239513 11299 raft_consensus.cc:729] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:24.239563 11301 raft_consensus.cc:348] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:24.239666 11298 raft_consensus.cc:348] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:24.240084 11298 raft_consensus.cc:374] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:24.240114 11301 raft_consensus.cc:374] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:24.240213 11301 raft_consensus.cc:729] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:24.240252 11298 raft_consensus.cc:729] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:24.240820 11299 consensus_queue.cc:260] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:24.240931 11298 consensus_queue.cc:260] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:24.240928 11301 consensus_queue.cc:260] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:24.243054 11300 ts_tablet_manager.cc:1398] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.026s user 0.019s sys 0.004s
I20240503 08:13:24.243642 11300 tablet_bootstrap.cc:492] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:24.245545 11300 tablet_bootstrap.cc:654] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:24.250460 11298 ts_tablet_manager.cc:1398] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.021s user 0.017s sys 0.003s
I20240503 08:13:24.251076 11298 tablet_bootstrap.cc:492] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:24.253471 11298 tablet_bootstrap.cc:654] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:24.256383 11299 ts_tablet_manager.cc:1398] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.030s user 0.014s sys 0.000s
I20240503 08:13:24.256376 11301 ts_tablet_manager.cc:1398] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.027s user 0.016s sys 0.005s
I20240503 08:13:24.259315 11298 tablet_bootstrap.cc:492] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:24.259599 11298 ts_tablet_manager.cc:1367] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 08:13:24.260041 11299 tablet_bootstrap.cc:492] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:24.260841 11298 raft_consensus.cc:348] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:24.261152 11298 raft_consensus.cc:374] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:24.261308 11298 raft_consensus.cc:729] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:24.261656 11298 consensus_queue.cc:260] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:24.262465 11298 ts_tablet_manager.cc:1398] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:24.262902 11301 tablet_bootstrap.cc:492] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:24.263432 11299 tablet_bootstrap.cc:654] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:24.265075 11301 tablet_bootstrap.cc:654] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:24.267752 11300 tablet_bootstrap.cc:492] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:24.267863 11299 tablet_bootstrap.cc:492] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:24.267926 11300 ts_tablet_manager.cc:1367] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.008s
I20240503 08:13:24.268036 11299 ts_tablet_manager.cc:1367] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240503 08:13:24.268059 11301 tablet_bootstrap.cc:492] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:24.268203 11301 ts_tablet_manager.cc:1367] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:13:24.268877 11300 raft_consensus.cc:348] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:24.269058 11300 raft_consensus.cc:374] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:24.269155 11300 raft_consensus.cc:729] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:24.269096 11299 raft_consensus.cc:348] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:24.269114 11301 raft_consensus.cc:348] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:24.269310 11301 raft_consensus.cc:374] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:24.269310 11299 raft_consensus.cc:374] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:24.269397 11299 raft_consensus.cc:729] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:24.269397 11301 raft_consensus.cc:729] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:24.269387 11300 consensus_queue.cc:260] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:24.269641 11301 consensus_queue.cc:260] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:24.269699 11299 consensus_queue.cc:260] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:24.269829 11300 ts_tablet_manager.cc:1398] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:24.270226 11301 ts_tablet_manager.cc:1398] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:24.270565 11301 tablet_bootstrap.cc:492] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:24.272163 11299 ts_tablet_manager.cc:1398] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:24.272342 11301 tablet_bootstrap.cc:654] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:24.281844 11301 tablet_bootstrap.cc:492] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:24.282049 11301 ts_tablet_manager.cc:1367] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
W20240503 08:13:24.282944 10355 tablet.cc:2369] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:13:24.282888 11301 raft_consensus.cc:348] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:24.283272 11301 raft_consensus.cc:374] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:24.283416 11301 raft_consensus.cc:729] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:24.283697 11301 consensus_queue.cc:260] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:24.284325 11301 ts_tablet_manager.cc:1398] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 08:13:24.304663 10747 tablet.cc:2369] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:13:24.306378 10486 tablet.cc:2369] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 08:13:24.309332 10617 tablet.cc:2369] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:13:24.333273 11307 raft_consensus.cc:482] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:24.333580 11307 raft_consensus.cc:504] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:24.335294 11307 leader_election.cc:288] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:24.343217 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:24.343253 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:24.343792 10309 raft_consensus.cc:2454] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 0.
I20240503 08:13:24.343796 10571 raft_consensus.cc:2454] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 0.
I20240503 08:13:24.344518 10376 leader_election.cc:302] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:24.344970 11307 raft_consensus.cc:2781] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:24.345137 11307 raft_consensus.cc:482] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:24.345245 11307 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:24.347471 11307 raft_consensus.cc:504] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:24.348107 11307 leader_election.cc:288] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:24.348757 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:24.348755 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:24.348992 10571 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:24.348992 10309 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:24.352286 10571 raft_consensus.cc:2454] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 1.
I20240503 08:13:24.352286 10309 raft_consensus.cc:2454] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 1.
I20240503 08:13:24.352891 10376 leader_election.cc:302] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:24.353286 11307 raft_consensus.cc:2781] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:24.353852 11307 raft_consensus.cc:686] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:24.354514 11307 consensus_queue.cc:237] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:24.359297 10166 catalog_manager.cc:5484] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 1 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:24.386318 11306 raft_consensus.cc:482] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:24.386600 11306 raft_consensus.cc:504] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:24.388597 11306 leader_election.cc:288] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:24.394806 11307 raft_consensus.cc:482] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:24.395185 11307 raft_consensus.cc:504] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:24.396629 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:24.396926 10309 raft_consensus.cc:2454] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 0.
I20240503 08:13:24.397042 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:24.397109 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:24.397387 10308 raft_consensus.cc:2454] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:24.397538 10374 leader_election.cc:302] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:24.397588 10440 raft_consensus.cc:2454] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:24.397976 11317 raft_consensus.cc:2781] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:24.398113 11307 leader_election.cc:288] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:24.398087 10635 leader_election.cc:302] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:24.398569 11306 raft_consensus.cc:2781] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:24.398627 11317 raft_consensus.cc:482] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:24.398731 11306 raft_consensus.cc:482] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:24.398794 11317 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:24.398842 11306 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:24.401921 11306 raft_consensus.cc:504] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:24.402458 11317 raft_consensus.cc:504] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:24.403288 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:24.403676 10440 raft_consensus.cc:2379] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 in current term 1: Already voted for candidate 2b3a6ff465524577a400de6af7e70f71 in this term.
I20240503 08:13:24.403906 11306 leader_election.cc:288] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:24.403659 11317 leader_election.cc:288] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:24.404263 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:24.404480 10308 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:24.404973 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:24.405272 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:24.405368 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:24.405984 10701 raft_consensus.cc:2379] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 in current term 1: Already voted for candidate 54534138f6814b809a25c03a46e5f931 in this term.
I20240503 08:13:24.406471 10373 leader_election.cc:302] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71; no voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:24.406790 11317 raft_consensus.cc:2726] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 08:13:24.407292 10308 raft_consensus.cc:2454] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:24.407712 10635 leader_election.cc:302] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:24.408046 11306 raft_consensus.cc:2781] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:24.408519 11306 raft_consensus.cc:686] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:24.409054 11306 consensus_queue.cc:237] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:24.413909 10166 catalog_manager.cc:5484] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:24.638208 11306 raft_consensus.cc:482] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:24.638458 11306 raft_consensus.cc:504] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:24.639240 11306 leader_election.cc:288] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:24.639900 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:24.640168 10308 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:24.640664 10635 leader_election.cc:302] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:24.641572 11306 raft_consensus.cc:2781] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:24.641685 11306 raft_consensus.cc:482] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:24.641774 11306 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:24.642232 11308 raft_consensus.cc:482] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:24.642454 11308 raft_consensus.cc:504] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:24.644809 11308 leader_election.cc:288] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:24.645027 11306 raft_consensus.cc:504] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:24.646078 11306 leader_election.cc:288] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:24.646485 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:24.646778 10308 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:24.651540 10308 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:24.654382 10635 leader_election.cc:302] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:24.655481 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:24.655766 10571 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:24.656131 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:24.656374 10570 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:24.660754 10570 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:24.661298 11306 raft_consensus.cc:2781] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:24.661599 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:24.661710 11306 raft_consensus.cc:686] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:24.662034 10308 raft_consensus.cc:2379] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 in current term 1: Already voted for candidate 54534138f6814b809a25c03a46e5f931 in this term.
I20240503 08:13:24.662079 11306 consensus_queue.cc:237] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:24.663843 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:24.664606 10504 leader_election.cc:302] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2; no voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:24.665027 11308 raft_consensus.cc:2726] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:13:24.665529 10166 catalog_manager.cc:5484] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:24.786701 11317 consensus_queue.cc:1035] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:24.797243 11317 consensus_queue.cc:1035] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:24.956561 11306 consensus_queue.cc:1035] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:24.965759 11335 consensus_queue.cc:1035] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:25.064239 11335 consensus_queue.cc:1035] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:25.069790 11335 consensus_queue.cc:1035] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:25.102638 11326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:25.103053 11326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:25.111032 11326 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 249.813 ms
time per row: 0.124906 ms
-----------------------------------------------------
Suppressions used:
count bytes template
14 3624 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:13:26.452595 10161 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44620:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:26.463043 10551 tablet_service.cc:1467] Processing CreateTablet for tablet cb392d6697d1442d86d3a64f956e8211 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d96e66a83b634ccfae2f08cf3e907c05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:26.463640 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb392d6697d1442d86d3a64f956e8211. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:26.463722 10681 tablet_service.cc:1467] Processing CreateTablet for tablet cb392d6697d1442d86d3a64f956e8211 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d96e66a83b634ccfae2f08cf3e907c05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:26.464174 10681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb392d6697d1442d86d3a64f956e8211. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:26.464143 10419 tablet_service.cc:1467] Processing CreateTablet for tablet 54e6601a404c46248232c7193aacaecb (DEFAULT_TABLE table=rebalance_test_table_1 [id=d96e66a83b634ccfae2f08cf3e907c05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:26.464569 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e6601a404c46248232c7193aacaecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:26.464687 10288 tablet_service.cc:1467] Processing CreateTablet for tablet b411562acf4c46c3b395d2901cf76a5a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d96e66a83b634ccfae2f08cf3e907c05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:26.465116 10288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b411562acf4c46c3b395d2901cf76a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:26.467772 10420 tablet_service.cc:1467] Processing CreateTablet for tablet cb392d6697d1442d86d3a64f956e8211 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d96e66a83b634ccfae2f08cf3e907c05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:26.468266 10420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb392d6697d1442d86d3a64f956e8211. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:26.471706 10680 tablet_service.cc:1467] Processing CreateTablet for tablet 54e6601a404c46248232c7193aacaecb (DEFAULT_TABLE table=rebalance_test_table_1 [id=d96e66a83b634ccfae2f08cf3e907c05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:26.472193 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e6601a404c46248232c7193aacaecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:26.476404 10550 tablet_service.cc:1467] Processing CreateTablet for tablet 54e6601a404c46248232c7193aacaecb (DEFAULT_TABLE table=rebalance_test_table_1 [id=d96e66a83b634ccfae2f08cf3e907c05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:26.476962 10550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e6601a404c46248232c7193aacaecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:26.480227 10418 tablet_service.cc:1467] Processing CreateTablet for tablet b411562acf4c46c3b395d2901cf76a5a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d96e66a83b634ccfae2f08cf3e907c05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:26.480510 10549 tablet_service.cc:1467] Processing CreateTablet for tablet b411562acf4c46c3b395d2901cf76a5a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d96e66a83b634ccfae2f08cf3e907c05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:26.480762 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b411562acf4c46c3b395d2901cf76a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:26.480963 10549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b411562acf4c46c3b395d2901cf76a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:26.484874 11389 tablet_bootstrap.cc:492] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:26.487216 11387 tablet_bootstrap.cc:492] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:26.488443 11389 tablet_bootstrap.cc:654] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:26.488915 11386 tablet_bootstrap.cc:492] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:26.489408 11388 tablet_bootstrap.cc:492] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:26.490726 11387 tablet_bootstrap.cc:654] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:26.491150 11388 tablet_bootstrap.cc:654] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:26.492511 11386 tablet_bootstrap.cc:654] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:26.497303 11389 tablet_bootstrap.cc:492] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:26.497673 11389 ts_tablet_manager.cc:1367] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 08:13:26.498909 11389 raft_consensus.cc:348] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.499079 11386 tablet_bootstrap.cc:492] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:26.499343 11389 raft_consensus.cc:374] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:26.499370 11386 ts_tablet_manager.cc:1367] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240503 08:13:26.499526 11389 raft_consensus.cc:729] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:26.499879 11389 consensus_queue.cc:260] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.500494 11386 raft_consensus.cc:348] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.500814 11386 raft_consensus.cc:374] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:26.500939 11386 raft_consensus.cc:729] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:26.501319 11386 consensus_queue.cc:260] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.506644 11386 ts_tablet_manager.cc:1398] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:13:26.507661 11389 ts_tablet_manager.cc:1398] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240503 08:13:26.508201 11389 tablet_bootstrap.cc:492] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:26.510710 11389 tablet_bootstrap.cc:654] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:26.512264 11386 tablet_bootstrap.cc:492] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:26.517163 11386 tablet_bootstrap.cc:654] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:26.517642 11387 tablet_bootstrap.cc:492] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:26.517815 11387 ts_tablet_manager.cc:1367] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.013s
I20240503 08:13:26.517907 11388 tablet_bootstrap.cc:492] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:26.518157 11388 ts_tablet_manager.cc:1367] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.012s
I20240503 08:13:26.518641 11389 tablet_bootstrap.cc:492] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:26.518894 11389 ts_tablet_manager.cc:1367] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 08:13:26.518826 11387 raft_consensus.cc:348] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.519116 11387 raft_consensus.cc:374] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:26.519272 11387 raft_consensus.cc:729] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:26.519491 11388 raft_consensus.cc:348] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:26.519675 11387 consensus_queue.cc:260] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.519860 11389 raft_consensus.cc:348] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.519991 11388 raft_consensus.cc:374] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:26.520145 11388 raft_consensus.cc:729] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:26.520167 11389 raft_consensus.cc:374] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:26.520249 11389 raft_consensus.cc:729] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:26.520577 11389 consensus_queue.cc:260] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.520520 11388 consensus_queue.cc:260] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:26.521363 11389 ts_tablet_manager.cc:1398] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:26.521797 11389 tablet_bootstrap.cc:492] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:26.523923 11389 tablet_bootstrap.cc:654] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:26.523862 11387 ts_tablet_manager.cc:1398] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 08:13:26.525933 11387 tablet_bootstrap.cc:492] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:26.528869 11386 tablet_bootstrap.cc:492] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:26.529167 11386 ts_tablet_manager.cc:1367] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20240503 08:13:26.529325 11387 tablet_bootstrap.cc:654] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:26.529573 11388 ts_tablet_manager.cc:1398] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20240503 08:13:26.530206 11386 raft_consensus.cc:348] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.530534 11386 raft_consensus.cc:374] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:26.530632 11386 raft_consensus.cc:729] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:26.530942 11386 consensus_queue.cc:260] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.531371 11389 tablet_bootstrap.cc:492] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:26.531447 11361 raft_consensus.cc:482] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:26.531600 11389 ts_tablet_manager.cc:1367] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 08:13:26.531621 11386 ts_tablet_manager.cc:1398] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:26.531616 11361 raft_consensus.cc:504] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.532337 11361 leader_election.cc:288] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:26.532553 11389 raft_consensus.cc:348] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:26.532764 11389 raft_consensus.cc:374] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:26.532857 11389 raft_consensus.cc:729] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:26.533042 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:26.533107 11389 consensus_queue.cc:260] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:26.533282 10701 raft_consensus.cc:2454] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 0.
I20240503 08:13:26.533857 11389 ts_tablet_manager.cc:1398] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:26.534190 10373 leader_election.cc:302] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:26.534497 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:26.534761 10570 raft_consensus.cc:2454] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 0.
I20240503 08:13:26.535795 11361 raft_consensus.cc:2781] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:26.535943 11361 raft_consensus.cc:482] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:26.536041 11361 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:26.539280 11361 raft_consensus.cc:504] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.540071 11387 tablet_bootstrap.cc:492] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:26.540258 11387 ts_tablet_manager.cc:1367] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240503 08:13:26.540500 11361 leader_election.cc:288] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:26.540577 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:26.540818 10701 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:26.540977 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:26.541082 11387 raft_consensus.cc:348] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.541222 10570 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:26.541304 11387 raft_consensus.cc:374] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:26.541375 11387 raft_consensus.cc:729] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:26.541632 11387 consensus_queue.cc:260] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.542160 11387 ts_tablet_manager.cc:1398] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:26.542999 11387 tablet_bootstrap.cc:492] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:26.543840 10701 raft_consensus.cc:2454] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 1.
I20240503 08:13:26.543862 10570 raft_consensus.cc:2454] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 1.
I20240503 08:13:26.544450 10376 leader_election.cc:302] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:26.544853 11361 raft_consensus.cc:2781] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:26.544890 11387 tablet_bootstrap.cc:654] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:26.545892 11361 raft_consensus.cc:686] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:26.546345 11361 consensus_queue.cc:237] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.547245 11387 tablet_bootstrap.cc:492] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:26.547432 11387 ts_tablet_manager.cc:1367] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:13:26.548362 11387 raft_consensus.cc:348] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:26.548652 11387 raft_consensus.cc:374] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:26.548753 11387 raft_consensus.cc:729] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:26.548982 11387 consensus_queue.cc:260] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:26.549427 11387 ts_tablet_manager.cc:1398] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:26.549517 10162 catalog_manager.cc:5484] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 1 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:26.617635 11306 raft_consensus.cc:482] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:26.617863 11306 raft_consensus.cc:504] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.618556 11306 leader_election.cc:288] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:26.619148 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:26.619261 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:26.619422 10570 raft_consensus.cc:2454] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:26.619530 10440 raft_consensus.cc:2454] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:26.619947 10637 leader_election.cc:302] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:26.620239 11306 raft_consensus.cc:2781] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:26.620330 11306 raft_consensus.cc:482] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:26.620430 11306 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:26.622571 11306 raft_consensus.cc:504] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.623574 11306 leader_election.cc:288] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:26.623832 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:26.624070 10570 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:26.624063 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:26.624248 10440 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:26.626226 10570 raft_consensus.cc:2454] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:26.626336 10440 raft_consensus.cc:2454] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:26.626814 10637 leader_election.cc:302] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:26.627164 11306 raft_consensus.cc:2781] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:26.627605 11306 raft_consensus.cc:686] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:26.627887 11306 consensus_queue.cc:237] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:26.630865 10162 catalog_manager.cc:5484] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:26.767994 11398 raft_consensus.cc:482] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:26.768214 11398 raft_consensus.cc:504] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:26.769763 11398 leader_election.cc:288] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:26.777653 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:26.777957 10440 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:26.778337 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:26.778517 10245 leader_election.cc:302] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:26.778609 10570 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:26.778926 11398 raft_consensus.cc:2781] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:26.779035 11398 raft_consensus.cc:482] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:26.779141 11398 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:26.781245 11398 raft_consensus.cc:504] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:26.781877 11398 leader_election.cc:288] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:26.782464 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:26.782480 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:26.782716 10440 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:26.782716 10570 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:26.785763 10570 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:26.785763 10440 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:26.786280 10245 leader_election.cc:302] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:26.786657 11398 raft_consensus.cc:2781] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:26.787072 11398 raft_consensus.cc:686] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:26.787575 11398 consensus_queue.cc:237] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:26.791560 10162 catalog_manager.cc:5484] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 1 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:27.037695 11306 consensus_queue.cc:1035] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:27.037734 11361 consensus_queue.cc:1035] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:27.044514 11403 consensus_queue.cc:1035] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:27.050333 11361 consensus_queue.cc:1035] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:27.225121 11411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:27.225473 11411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:27.233132 11411 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 08:13:27.283849 11398 consensus_queue.cc:1035] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:27.293002 11445 consensus_queue.cc:1035] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 239.006 ms
time per row: 0.119503 ms
I20240503 08:13:27.629840 10166 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44620:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:27.639714 10680 tablet_service.cc:1467] Processing CreateTablet for tablet 205f68b8cf6d4b948576765252160224 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f1b69ea4982427ba1536e445418f288]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:27.640069 10419 tablet_service.cc:1467] Processing CreateTablet for tablet 205f68b8cf6d4b948576765252160224 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f1b69ea4982427ba1536e445418f288]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:27.640275 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205f68b8cf6d4b948576765252160224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.640524 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205f68b8cf6d4b948576765252160224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.640937 10549 tablet_service.cc:1467] Processing CreateTablet for tablet 4764c2955533410188d40d9465156bcc (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f1b69ea4982427ba1536e445418f288]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:27.641392 10549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764c2955533410188d40d9465156bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.641325 10288 tablet_service.cc:1467] Processing CreateTablet for tablet 205f68b8cf6d4b948576765252160224 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f1b69ea4982427ba1536e445418f288]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:27.641729 10288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205f68b8cf6d4b948576765252160224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.649289 10289 tablet_service.cc:1467] Processing CreateTablet for tablet 4764c2955533410188d40d9465156bcc (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f1b69ea4982427ba1536e445418f288]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:27.649791 10289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764c2955533410188d40d9465156bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.650859 10287 tablet_service.cc:1467] Processing CreateTablet for tablet 6c636ced06a44624910078f2e75eea2f (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f1b69ea4982427ba1536e445418f288]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:27.652537 10681 tablet_service.cc:1467] Processing CreateTablet for tablet 4764c2955533410188d40d9465156bcc (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f1b69ea4982427ba1536e445418f288]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:27.652585 10418 tablet_service.cc:1467] Processing CreateTablet for tablet 6c636ced06a44624910078f2e75eea2f (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f1b69ea4982427ba1536e445418f288]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:27.653138 10681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764c2955533410188d40d9465156bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.653175 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c636ced06a44624910078f2e75eea2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.653630 11482 tablet_bootstrap.cc:492] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:27.654497 11479 tablet_bootstrap.cc:492] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:27.656520 11479 tablet_bootstrap.cc:654] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:27.657202 10679 tablet_service.cc:1467] Processing CreateTablet for tablet 6c636ced06a44624910078f2e75eea2f (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f1b69ea4982427ba1536e445418f288]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:27.657745 10679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c636ced06a44624910078f2e75eea2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.658141 11480 tablet_bootstrap.cc:492] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:27.660270 11480 tablet_bootstrap.cc:654] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:27.663573 10287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c636ced06a44624910078f2e75eea2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:27.665798 11482 tablet_bootstrap.cc:654] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:27.674458 11482 tablet_bootstrap.cc:492] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:27.674750 11482 ts_tablet_manager.cc:1367] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20240503 08:13:27.675737 11482 raft_consensus.cc:348] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.676056 11482 raft_consensus.cc:374] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:27.676188 11482 raft_consensus.cc:729] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:27.676519 11482 consensus_queue.cc:260] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.677381 11482 ts_tablet_manager.cc:1398] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:27.677978 11482 tablet_bootstrap.cc:492] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:27.679257 11482 tablet_bootstrap.cc:654] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:27.681102 11481 tablet_bootstrap.cc:492] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:27.681614 11482 tablet_bootstrap.cc:492] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:27.681829 11482 ts_tablet_manager.cc:1367] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:27.682756 11482 raft_consensus.cc:348] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:27.683136 11482 raft_consensus.cc:374] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:27.683207 11481 tablet_bootstrap.cc:654] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:27.683528 11482 raft_consensus.cc:729] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:27.683873 11482 consensus_queue.cc:260] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:27.684679 11479 tablet_bootstrap.cc:492] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:27.684679 11482 ts_tablet_manager.cc:1398] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:27.684842 11479 ts_tablet_manager.cc:1367] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20240503 08:13:27.685089 11482 tablet_bootstrap.cc:492] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:27.685686 11479 raft_consensus.cc:348] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.685906 11479 raft_consensus.cc:374] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:27.686007 11479 raft_consensus.cc:729] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:27.686326 11479 consensus_queue.cc:260] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.687450 11482 tablet_bootstrap.cc:654] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:27.688222 11479 ts_tablet_manager.cc:1398] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:27.688532 11479 tablet_bootstrap.cc:492] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:27.691251 11480 tablet_bootstrap.cc:492] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:27.691504 11480 ts_tablet_manager.cc:1367] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.004s
I20240503 08:13:27.692488 11480 raft_consensus.cc:348] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.692718 11480 raft_consensus.cc:374] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:27.692811 11480 raft_consensus.cc:729] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:27.693094 11480 consensus_queue.cc:260] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.693866 11480 ts_tablet_manager.cc:1398] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:27.694221 11480 tablet_bootstrap.cc:492] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:27.695999 11480 tablet_bootstrap.cc:654] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:27.696442 11482 tablet_bootstrap.cc:492] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:27.696672 11482 ts_tablet_manager.cc:1367] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20240503 08:13:27.697599 11482 raft_consensus.cc:348] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.697885 11482 raft_consensus.cc:374] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:27.698036 11482 raft_consensus.cc:729] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:27.698364 11482 consensus_queue.cc:260] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.699504 11479 tablet_bootstrap.cc:654] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:27.700275 11481 tablet_bootstrap.cc:492] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:27.700449 11481 ts_tablet_manager.cc:1367] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.005s
I20240503 08:13:27.701404 11482 ts_tablet_manager.cc:1398] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 08:13:27.702296 11481 raft_consensus.cc:348] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:27.702483 11481 raft_consensus.cc:374] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:27.702576 11481 raft_consensus.cc:729] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:27.702824 11481 consensus_queue.cc:260] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:27.703657 11481 ts_tablet_manager.cc:1398] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 08:13:27.704762 11480 tablet_bootstrap.cc:492] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:27.704980 11480 ts_tablet_manager.cc:1367] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 08:13:27.705888 11480 raft_consensus.cc:348] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:27.706137 11480 raft_consensus.cc:374] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:27.706250 11480 raft_consensus.cc:729] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:27.706547 11480 consensus_queue.cc:260] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:27.707144 11480 ts_tablet_manager.cc:1398] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:27.707644 11480 tablet_bootstrap.cc:492] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:27.709349 11480 tablet_bootstrap.cc:654] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:27.710482 11479 tablet_bootstrap.cc:492] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:27.711581 11479 ts_tablet_manager.cc:1367] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240503 08:13:27.712724 11480 tablet_bootstrap.cc:492] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:27.712627 11479 raft_consensus.cc:348] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.712870 11479 raft_consensus.cc:374] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:27.712877 11480 ts_tablet_manager.cc:1367] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 08:13:27.712968 11479 raft_consensus.cc:729] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:27.713234 11479 consensus_queue.cc:260] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.713572 11480 raft_consensus.cc:348] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.713712 11479 ts_tablet_manager.cc:1398] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:27.713783 11480 raft_consensus.cc:374] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:27.713932 11480 raft_consensus.cc:729] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:27.714174 11480 consensus_queue.cc:260] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.714700 11480 ts_tablet_manager.cc:1398] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:27.726007 11443 raft_consensus.cc:482] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:27.726179 11443 raft_consensus.cc:504] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.727020 11443 leader_election.cc:288] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:27.727533 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:27.727797 10440 raft_consensus.cc:2454] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:27.728279 10245 leader_election.cc:302] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:27.728629 11443 raft_consensus.cc:2781] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:27.728780 11443 raft_consensus.cc:482] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:27.728895 11443 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:27.731449 11443 raft_consensus.cc:504] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.732244 11443 leader_election.cc:288] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:27.732856 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:27.733111 10440 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:27.734676 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:27.734699 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:27.734962 10701 raft_consensus.cc:2454] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:27.735524 10440 raft_consensus.cc:2454] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:27.735936 10245 leader_election.cc:302] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 54534138f6814b809a25c03a46e5f931
I20240503 08:13:27.736308 11443 raft_consensus.cc:2781] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:27.736464 11443 raft_consensus.cc:686] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:27.736783 11443 consensus_queue.cc:237] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.739761 10164 catalog_manager.cc:5484] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 1 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:27.757464 11474 raft_consensus.cc:482] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:27.757669 11474 raft_consensus.cc:504] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.758579 11474 leader_election.cc:288] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:27.759238 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:27.759243 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:27.759526 10700 raft_consensus.cc:2454] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 0.
I20240503 08:13:27.759526 10309 raft_consensus.cc:2454] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 0.
I20240503 08:13:27.760005 10374 leader_election.cc:302] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:27.760362 11474 raft_consensus.cc:2781] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:27.760468 11474 raft_consensus.cc:482] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:27.760556 11474 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:27.762764 11474 raft_consensus.cc:504] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.763401 11474 leader_election.cc:288] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:27.764060 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:27.764060 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:27.764294 10700 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:27.764294 10309 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:27.767381 10700 raft_consensus.cc:2454] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 1.
I20240503 08:13:27.767354 10309 raft_consensus.cc:2454] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 1.
I20240503 08:13:27.768028 10374 leader_election.cc:302] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:27.768435 11474 raft_consensus.cc:2781] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:27.768595 11474 raft_consensus.cc:686] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:27.768888 11474 consensus_queue.cc:237] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:27.772064 10164 catalog_manager.cc:5484] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 1 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:27.779498 11413 raft_consensus.cc:482] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:27.779662 11413 raft_consensus.cc:504] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:27.780462 11413 leader_election.cc:288] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:27.780957 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:27.780982 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:27.781193 10309 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 0.
I20240503 08:13:27.781234 10700 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 0.
I20240503 08:13:27.781635 10505 leader_election.cc:302] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:27.781936 11413 raft_consensus.cc:2781] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:27.782038 11413 raft_consensus.cc:482] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:27.782135 11413 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:27.784216 11413 raft_consensus.cc:504] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:27.784915 11413 leader_election.cc:288] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:27.785463 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:27.785586 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:27.785717 10309 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:27.785842 10700 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:27.787950 10309 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 1.
I20240503 08:13:27.788081 10700 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 1.
I20240503 08:13:27.788461 10505 leader_election.cc:302] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:27.788813 11413 raft_consensus.cc:2781] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:27.789408 11413 raft_consensus.cc:686] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:27.789928 11413 consensus_queue.cc:237] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:27.793983 10164 catalog_manager.cc:5484] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 1 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
W20240503 08:13:28.181703 11497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:28.182049 11497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:28.190203 11497 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:28.192600 11413 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:28.200191 11413 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 08:13:28.276242 11474 consensus_queue.cc:1035] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:28.280275 11443 consensus_queue.cc:1035] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:28.281397 10701 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:28.288357 11443 consensus_queue.cc:1035] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:28.296113 11474 consensus_queue.cc:1035] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 347.97 ms
time per row: 0.173985 ms
I20240503 08:13:28.707353 10161 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44620:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:28.716868 10287 tablet_service.cc:1467] Processing CreateTablet for tablet 835ed4599b904cd695e171afe6e65efa (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ce01637b43b487abe6bcb5f7a1b8767]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:28.717038 10549 tablet_service.cc:1467] Processing CreateTablet for tablet 835ed4599b904cd695e171afe6e65efa (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ce01637b43b487abe6bcb5f7a1b8767]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:28.717450 10287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835ed4599b904cd695e171afe6e65efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:28.717505 10549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835ed4599b904cd695e171afe6e65efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:28.718050 10289 tablet_service.cc:1467] Processing CreateTablet for tablet 22b825c0acaa453f837aa7ca7a5543b0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ce01637b43b487abe6bcb5f7a1b8767]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:28.718466 10289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b825c0acaa453f837aa7ca7a5543b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:28.718923 10680 tablet_service.cc:1467] Processing CreateTablet for tablet 22b825c0acaa453f837aa7ca7a5543b0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ce01637b43b487abe6bcb5f7a1b8767]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:28.719381 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b825c0acaa453f837aa7ca7a5543b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:28.721755 10418 tablet_service.cc:1467] Processing CreateTablet for tablet 835ed4599b904cd695e171afe6e65efa (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ce01637b43b487abe6bcb5f7a1b8767]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:28.722247 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835ed4599b904cd695e171afe6e65efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:28.724692 10679 tablet_service.cc:1467] Processing CreateTablet for tablet f6443a176ba74f649ecc8da8717ed879 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ce01637b43b487abe6bcb5f7a1b8767]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:28.725137 10679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6443a176ba74f649ecc8da8717ed879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:28.728425 10551 tablet_service.cc:1467] Processing CreateTablet for tablet f6443a176ba74f649ecc8da8717ed879 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ce01637b43b487abe6bcb5f7a1b8767]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:28.728446 10550 tablet_service.cc:1467] Processing CreateTablet for tablet 22b825c0acaa453f837aa7ca7a5543b0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ce01637b43b487abe6bcb5f7a1b8767]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:28.728694 10419 tablet_service.cc:1467] Processing CreateTablet for tablet f6443a176ba74f649ecc8da8717ed879 (DEFAULT_TABLE table=rebalance_test_table_3 [id=0ce01637b43b487abe6bcb5f7a1b8767]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:28.728941 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6443a176ba74f649ecc8da8717ed879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:28.729125 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6443a176ba74f649ecc8da8717ed879. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:28.729138 10550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b825c0acaa453f837aa7ca7a5543b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:28.736027 11569 tablet_bootstrap.cc:492] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:28.738831 11569 tablet_bootstrap.cc:654] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:28.743387 11570 tablet_bootstrap.cc:492] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:28.746675 11570 tablet_bootstrap.cc:654] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:28.747895 11572 tablet_bootstrap.cc:492] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:28.749449 11568 tablet_bootstrap.cc:492] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:28.749720 11569 tablet_bootstrap.cc:492] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:28.749902 11569 ts_tablet_manager.cc:1367] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240503 08:13:28.750420 11572 tablet_bootstrap.cc:654] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:28.750828 11570 tablet_bootstrap.cc:492] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:28.750790 11569 raft_consensus.cc:348] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:28.751024 11569 raft_consensus.cc:374] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:28.751075 11570 ts_tablet_manager.cc:1367] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240503 08:13:28.751128 11569 raft_consensus.cc:729] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:28.751412 11569 consensus_queue.cc:260] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:28.752035 11569 ts_tablet_manager.cc:1398] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:28.752399 11569 tablet_bootstrap.cc:492] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:28.752952 11570 raft_consensus.cc:348] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:28.753233 11570 raft_consensus.cc:374] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:28.753363 11570 raft_consensus.cc:729] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:28.753623 11570 consensus_queue.cc:260] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:28.754062 11569 tablet_bootstrap.cc:654] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:28.754331 11570 ts_tablet_manager.cc:1398] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 08:13:28.755510 11570 tablet_bootstrap.cc:492] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:28.757671 11570 tablet_bootstrap.cc:654] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:28.759732 11572 tablet_bootstrap.cc:492] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:28.759912 11572 ts_tablet_manager.cc:1367] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 08:13:28.760742 11568 tablet_bootstrap.cc:654] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:28.760798 11572 raft_consensus.cc:348] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:28.761010 11572 raft_consensus.cc:374] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:28.761103 11572 raft_consensus.cc:729] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:28.761358 11572 consensus_queue.cc:260] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:28.761911 11572 ts_tablet_manager.cc:1398] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:28.764400 11572 tablet_bootstrap.cc:492] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:28.765985 11306 raft_consensus.cc:482] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:28.766242 11572 tablet_bootstrap.cc:654] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:28.766201 11306 raft_consensus.cc:504] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:28.766980 11306 leader_election.cc:288] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:28.767935 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:28.768667 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:28.769641 11570 tablet_bootstrap.cc:492] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:28.769944 11570 ts_tablet_manager.cc:1367] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
W20240503 08:13:28.770104 10635 leader_election.cc:341] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:28.771271 11570 raft_consensus.cc:348] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:28.771822 11570 raft_consensus.cc:374] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:28.771984 11570 raft_consensus.cc:729] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:28.772392 11570 consensus_queue.cc:260] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:28.773284 11570 ts_tablet_manager.cc:1398] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240503 08:13:28.774327 10637 leader_election.cc:341] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:13:28.774513 10637 leader_election.cc:302] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54534138f6814b809a25c03a46e5f931; no voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:28.775261 11306 raft_consensus.cc:2726] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:13:28.780201 11569 tablet_bootstrap.cc:492] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:28.780431 11569 ts_tablet_manager.cc:1367] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.009s
I20240503 08:13:28.781132 11572 tablet_bootstrap.cc:492] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:28.781355 11568 tablet_bootstrap.cc:492] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:28.781347 11572 ts_tablet_manager.cc:1367] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20240503 08:13:28.781318 11569 raft_consensus.cc:348] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:28.781589 11569 raft_consensus.cc:374] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:28.781649 11568 ts_tablet_manager.cc:1367] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20240503 08:13:28.781682 11569 raft_consensus.cc:729] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:28.781960 11569 consensus_queue.cc:260] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:28.782550 11569 ts_tablet_manager.cc:1398] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:28.782886 11569 tablet_bootstrap.cc:492] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:28.782765 11568 raft_consensus.cc:348] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:28.782969 11568 raft_consensus.cc:374] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:28.783053 11568 raft_consensus.cc:729] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:28.783202 11572 raft_consensus.cc:348] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:28.783491 11572 raft_consensus.cc:374] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:28.783380 11568 consensus_queue.cc:260] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:28.783577 11572 raft_consensus.cc:729] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:28.783967 11568 ts_tablet_manager.cc:1398] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:28.783857 11572 consensus_queue.cc:260] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:28.784265 11568 tablet_bootstrap.cc:492] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:28.784700 11569 tablet_bootstrap.cc:654] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:28.786084 11568 tablet_bootstrap.cc:654] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:28.786396 11572 ts_tablet_manager.cc:1398] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:28.793089 11568 tablet_bootstrap.cc:492] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:28.793329 11568 ts_tablet_manager.cc:1367] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20240503 08:13:28.793367 11569 tablet_bootstrap.cc:492] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:28.793510 11569 ts_tablet_manager.cc:1367] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240503 08:13:28.794234 11568 raft_consensus.cc:348] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:28.794262 11569 raft_consensus.cc:348] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:28.794471 11569 raft_consensus.cc:374] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:28.794487 11568 raft_consensus.cc:374] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:28.794559 11568 raft_consensus.cc:729] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:28.794592 11569 raft_consensus.cc:729] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:28.794836 11569 consensus_queue.cc:260] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:28.794840 11568 consensus_queue.cc:260] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:28.795409 11568 ts_tablet_manager.cc:1398] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 08:13:28.795547 11569 ts_tablet_manager.cc:1398] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:13:28.807482 11443 raft_consensus.cc:482] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:28.807710 11443 raft_consensus.cc:504] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:28.808487 11443 leader_election.cc:288] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:28.809422 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:28.809422 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:28.809701 10569 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:28.809718 10440 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:28.810371 10245 leader_election.cc:302] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:28.810763 11443 raft_consensus.cc:2781] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:28.810870 11443 raft_consensus.cc:482] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:28.810972 11443 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:28.812983 11443 raft_consensus.cc:504] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:28.813668 11443 leader_election.cc:288] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:28.814734 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:28.815016 10569 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:28.817682 10569 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:28.818259 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:28.818521 10440 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:28.820904 10440 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:28.821419 10245 leader_election.cc:302] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:28.821835 11443 raft_consensus.cc:2781] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:28.822085 11443 raft_consensus.cc:686] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:28.822491 11443 consensus_queue.cc:237] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:28.826259 10161 catalog_manager.cc:5484] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 1 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:28.855279 11578 raft_consensus.cc:482] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:28.855556 11578 raft_consensus.cc:504] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:28.856341 11578 leader_election.cc:288] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:28.856971 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:28.857069 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:28.857180 10569 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:28.857311 10440 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:28.857631 10637 leader_election.cc:302] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:28.857904 11578 raft_consensus.cc:2781] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:28.858016 11578 raft_consensus.cc:482] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:28.858108 11578 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:28.860178 11578 raft_consensus.cc:504] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:28.860790 11578 leader_election.cc:288] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:28.861490 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:28.861490 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:28.861726 10440 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:28.861726 10569 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:28.864743 10440 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:28.864743 10569 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:28.865254 10637 leader_election.cc:302] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:28.865638 11578 raft_consensus.cc:2781] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:28.865789 11578 raft_consensus.cc:686] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:28.866070 11578 consensus_queue.cc:237] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:28.869151 10161 catalog_manager.cc:5484] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:28.962749 11443 raft_consensus.cc:482] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:28.963008 11443 raft_consensus.cc:504] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:28.963883 11443 leader_election.cc:288] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:28.964308 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:28.964354 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:28.964546 10569 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:28.964602 10701 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:28.965001 10245 leader_election.cc:302] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:28.965293 11443 raft_consensus.cc:2781] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:28.965411 11443 raft_consensus.cc:482] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:28.965524 11443 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:28.967440 11443 raft_consensus.cc:504] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:28.968044 11443 leader_election.cc:288] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:28.968590 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:28.968590 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:28.968794 10569 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:28.968808 10701 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:28.970772 10701 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:28.970772 10569 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:28.971254 10242 leader_election.cc:302] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:28.971675 11443 raft_consensus.cc:2781] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:28.971830 11443 raft_consensus.cc:686] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:28.972137 11443 consensus_queue.cc:237] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:28.975106 10161 catalog_manager.cc:5484] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 1 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:29.262231 11578 consensus_queue.cc:1035] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:29.269191 11588 consensus_queue.cc:1035] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:29.358696 11585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:29.359048 11585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:13:29.361884 11443 consensus_queue.cc:1035] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:29.368563 11443 consensus_queue.cc:1035] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:29.369796 11585 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:29.380183 11598 consensus_queue.cc:1035] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:29.387389 11443 consensus_queue.cc:1035] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 224.394 ms
time per row: 0.112197 ms
I20240503 08:13:29.790591 10168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44620:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:29.801268 10287 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8e6483973f420e97503f409130cc36 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3df55b9a55874103be7e84e4bc06a324]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:29.801265 10549 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8e6483973f420e97503f409130cc36 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3df55b9a55874103be7e84e4bc06a324]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:29.801774 10549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8e6483973f420e97503f409130cc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:29.801779 10287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8e6483973f420e97503f409130cc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:29.802963 10289 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3c45457bd446fb8dec0197c8e809ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=3df55b9a55874103be7e84e4bc06a324]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:29.803622 10289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3c45457bd446fb8dec0197c8e809ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:29.807826 10288 tablet_service.cc:1467] Processing CreateTablet for tablet 7d01df9a11104a33a7971e133310acce (DEFAULT_TABLE table=rebalance_test_table_4 [id=3df55b9a55874103be7e84e4bc06a324]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:29.808331 10288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d01df9a11104a33a7971e133310acce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:29.809566 10680 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3c45457bd446fb8dec0197c8e809ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=3df55b9a55874103be7e84e4bc06a324]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:29.810074 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3c45457bd446fb8dec0197c8e809ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:29.810664 10418 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8e6483973f420e97503f409130cc36 (DEFAULT_TABLE table=rebalance_test_table_4 [id=3df55b9a55874103be7e84e4bc06a324]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:29.811235 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8e6483973f420e97503f409130cc36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:29.812203 10550 tablet_service.cc:1467] Processing CreateTablet for tablet 7d01df9a11104a33a7971e133310acce (DEFAULT_TABLE table=rebalance_test_table_4 [id=3df55b9a55874103be7e84e4bc06a324]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:29.812623 10550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d01df9a11104a33a7971e133310acce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:29.814317 10419 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3c45457bd446fb8dec0197c8e809ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=3df55b9a55874103be7e84e4bc06a324]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:29.814864 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3c45457bd446fb8dec0197c8e809ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:29.815483 10679 tablet_service.cc:1467] Processing CreateTablet for tablet 7d01df9a11104a33a7971e133310acce (DEFAULT_TABLE table=rebalance_test_table_4 [id=3df55b9a55874103be7e84e4bc06a324]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:29.815932 10679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d01df9a11104a33a7971e133310acce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:29.823522 11648 tablet_bootstrap.cc:492] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:29.828949 11648 tablet_bootstrap.cc:654] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:29.839262 11648 tablet_bootstrap.cc:492] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:29.839486 11648 ts_tablet_manager.cc:1367] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 08:13:29.840380 11648 raft_consensus.cc:348] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.840620 11648 raft_consensus.cc:374] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:29.840734 11648 raft_consensus.cc:729] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:29.841013 11648 consensus_queue.cc:260] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.841730 11648 ts_tablet_manager.cc:1398] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:29.844318 11649 tablet_bootstrap.cc:492] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:29.845835 11652 tablet_bootstrap.cc:492] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:29.847996 11652 tablet_bootstrap.cc:654] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:29.848142 11650 tablet_bootstrap.cc:492] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:29.850180 11650 tablet_bootstrap.cc:654] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:29.853621 11648 tablet_bootstrap.cc:492] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:29.855469 11648 tablet_bootstrap.cc:654] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:29.857385 11650 tablet_bootstrap.cc:492] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:29.857668 11650 ts_tablet_manager.cc:1367] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240503 08:13:29.857836 11649 tablet_bootstrap.cc:654] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:29.858811 11650 raft_consensus.cc:348] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:29.859141 11650 raft_consensus.cc:374] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:29.859300 11650 raft_consensus.cc:729] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:29.859633 11652 tablet_bootstrap.cc:492] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:29.859870 11652 ts_tablet_manager.cc:1367] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240503 08:13:29.859750 11650 consensus_queue.cc:260] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:29.860575 11650 ts_tablet_manager.cc:1398] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:29.860976 11650 tablet_bootstrap.cc:492] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:29.861706 11652 raft_consensus.cc:348] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.861963 11652 raft_consensus.cc:374] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:29.862069 11652 raft_consensus.cc:729] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:29.862320 11652 consensus_queue.cc:260] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.862951 11652 ts_tablet_manager.cc:1398] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 08:13:29.863153 11650 tablet_bootstrap.cc:654] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:29.865109 11652 tablet_bootstrap.cc:492] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:29.869460 11652 tablet_bootstrap.cc:654] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:29.870358 11648 tablet_bootstrap.cc:492] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:29.870569 11648 ts_tablet_manager.cc:1367] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.006s
I20240503 08:13:29.871603 11649 tablet_bootstrap.cc:492] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:29.871507 11648 raft_consensus.cc:348] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:29.871779 11648 raft_consensus.cc:374] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:29.871774 11649 ts_tablet_manager.cc:1367] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240503 08:13:29.871870 11648 raft_consensus.cc:729] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:29.872143 11648 consensus_queue.cc:260] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:29.872735 11648 ts_tablet_manager.cc:1398] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:29.872680 11649 raft_consensus.cc:348] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.872898 11649 raft_consensus.cc:374] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:29.873000 11649 raft_consensus.cc:729] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:29.874982 11649 consensus_queue.cc:260] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.875201 11650 tablet_bootstrap.cc:492] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:29.875561 11650 ts_tablet_manager.cc:1367] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20240503 08:13:29.875728 11649 ts_tablet_manager.cc:1398] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 08:13:29.876065 11649 tablet_bootstrap.cc:492] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:29.876592 11650 raft_consensus.cc:348] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.876919 11650 raft_consensus.cc:374] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:29.877051 11650 raft_consensus.cc:729] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:29.877444 11650 consensus_queue.cc:260] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.877667 11652 tablet_bootstrap.cc:492] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:29.877879 11652 ts_tablet_manager.cc:1367] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20240503 08:13:29.878194 11650 ts_tablet_manager.cc:1398] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:29.878728 11652 raft_consensus.cc:348] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.878957 11652 raft_consensus.cc:374] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:29.879047 11652 raft_consensus.cc:729] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:29.879428 11649 tablet_bootstrap.cc:654] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:29.879549 11652 consensus_queue.cc:260] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.880311 11652 ts_tablet_manager.cc:1398] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 08:13:29.880621 11652 tablet_bootstrap.cc:492] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:29.882349 11652 tablet_bootstrap.cc:654] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:29.882936 11649 tablet_bootstrap.cc:492] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:29.883141 11649 ts_tablet_manager.cc:1367] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240503 08:13:29.883999 11649 raft_consensus.cc:348] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.884236 11649 raft_consensus.cc:374] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:29.884336 11649 raft_consensus.cc:729] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:29.884624 11649 consensus_queue.cc:260] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:29.885398 11649 ts_tablet_manager.cc:1398] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:29.886818 11652 tablet_bootstrap.cc:492] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:29.886950 11652 ts_tablet_manager.cc:1367] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240503 08:13:29.887812 11652 raft_consensus.cc:348] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:29.888018 11652 raft_consensus.cc:374] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:29.888141 11652 raft_consensus.cc:729] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:29.888429 11652 consensus_queue.cc:260] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:29.889089 11652 ts_tablet_manager.cc:1398] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:29.969115 11544 raft_consensus.cc:482] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:29.969314 11544 raft_consensus.cc:504] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:29.970079 11544 leader_election.cc:288] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:29.970772 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:29.970791 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:29.971047 10699 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 0.
I20240503 08:13:29.971073 10308 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 0.
I20240503 08:13:29.971622 10504 leader_election.cc:302] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:29.971993 11544 raft_consensus.cc:2781] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:29.972139 11544 raft_consensus.cc:482] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:29.972239 11544 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:29.974303 11544 raft_consensus.cc:504] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:29.974938 11544 leader_election.cc:288] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:29.975522 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:29.975544 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:29.975733 10699 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:29.975744 10308 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:29.977762 10308 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 1.
I20240503 08:13:29.977816 10699 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 1.
I20240503 08:13:29.978296 10505 leader_election.cc:302] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:29.978695 11544 raft_consensus.cc:2781] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:29.979236 11544 raft_consensus.cc:686] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:29.979616 11544 consensus_queue.cc:237] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:29.982502 10161 catalog_manager.cc:5484] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 1 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:30.006320 11598 raft_consensus.cc:482] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:30.006551 11598 raft_consensus.cc:504] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:30.007184 11598 leader_election.cc:288] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:30.007834 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:30.008057 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:30.008122 10571 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:30.008340 10440 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:30.008673 10245 leader_election.cc:302] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:30.009007 11598 raft_consensus.cc:2781] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:30.009120 11598 raft_consensus.cc:482] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:30.009246 11598 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:30.011921 11598 raft_consensus.cc:504] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:30.012550 11598 leader_election.cc:288] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:30.013125 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:30.013180 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:30.013347 10571 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:30.013406 10440 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:30.015771 10571 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:30.015772 10440 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:30.016294 10245 leader_election.cc:302] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:30.016728 11598 raft_consensus.cc:2781] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:30.016933 11598 raft_consensus.cc:686] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:30.017215 11598 consensus_queue.cc:237] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:30.020272 10161 catalog_manager.cc:5484] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 1 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:30.124552 11578 raft_consensus.cc:482] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:30.124755 11578 raft_consensus.cc:504] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:30.125460 11578 leader_election.cc:288] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:30.126078 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:30.126085 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:30.126302 10308 raft_consensus.cc:2454] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:30.126308 10440 raft_consensus.cc:2454] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:30.126791 10637 leader_election.cc:302] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:30.127142 11578 raft_consensus.cc:2781] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:30.127247 11578 raft_consensus.cc:482] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:30.127310 11578 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:30.129197 11578 raft_consensus.cc:504] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:30.129789 11578 leader_election.cc:288] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:30.130445 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:30.130451 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:30.130679 10440 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:30.130673 10308 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:30.132705 10440 raft_consensus.cc:2454] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:30.132721 10308 raft_consensus.cc:2454] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:30.133252 10637 leader_election.cc:302] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:30.133719 11578 raft_consensus.cc:2781] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:30.133896 11578 raft_consensus.cc:686] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:30.134212 11578 consensus_queue.cc:237] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:30.137027 10161 catalog_manager.cc:5484] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:30.406857 11598 consensus_queue.cc:1035] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:30.416635 11598 consensus_queue.cc:1035] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:30.486019 11544 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:30.492210 11673 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:30.559805 11665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:30.560139 11665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:30.568286 11665 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 08:13:30.685374 11578 consensus_queue.cc:1035] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:30.705365 11578 consensus_queue.cc:1035] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 326.839 ms
time per row: 0.163419 ms
I20240503 08:13:31.059772 10161 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44620:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:31.069105 10418 tablet_service.cc:1467] Processing CreateTablet for tablet a3a424b97c9b453081b568ab96bead7a (DEFAULT_TABLE table=rebalance_test_table_5 [id=5e1afceea1ff4a8191267514237b5e42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:31.069399 10550 tablet_service.cc:1467] Processing CreateTablet for tablet a3a424b97c9b453081b568ab96bead7a (DEFAULT_TABLE table=rebalance_test_table_5 [id=5e1afceea1ff4a8191267514237b5e42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:31.069622 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a424b97c9b453081b568ab96bead7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.069693 10679 tablet_service.cc:1467] Processing CreateTablet for tablet a3a424b97c9b453081b568ab96bead7a (DEFAULT_TABLE table=rebalance_test_table_5 [id=5e1afceea1ff4a8191267514237b5e42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:31.069836 10550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a424b97c9b453081b568ab96bead7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.070135 10679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a424b97c9b453081b568ab96bead7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.072580 10287 tablet_service.cc:1467] Processing CreateTablet for tablet 68032d6c4ad949edbf1bb55aea7f5562 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5e1afceea1ff4a8191267514237b5e42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:31.073071 10287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68032d6c4ad949edbf1bb55aea7f5562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.073535 10549 tablet_service.cc:1467] Processing CreateTablet for tablet ab9203ab3c96465682075b911880b320 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5e1afceea1ff4a8191267514237b5e42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:31.073971 10549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9203ab3c96465682075b911880b320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.074342 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 68032d6c4ad949edbf1bb55aea7f5562 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5e1afceea1ff4a8191267514237b5e42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:31.075057 10680 tablet_service.cc:1467] Processing CreateTablet for tablet ab9203ab3c96465682075b911880b320 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5e1afceea1ff4a8191267514237b5e42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:31.075379 10419 tablet_service.cc:1467] Processing CreateTablet for tablet ab9203ab3c96465682075b911880b320 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5e1afceea1ff4a8191267514237b5e42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:31.075520 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9203ab3c96465682075b911880b320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.075511 10420 tablet_service.cc:1467] Processing CreateTablet for tablet 68032d6c4ad949edbf1bb55aea7f5562 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5e1afceea1ff4a8191267514237b5e42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:31.075789 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9203ab3c96465682075b911880b320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.076107 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68032d6c4ad949edbf1bb55aea7f5562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.083369 10420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68032d6c4ad949edbf1bb55aea7f5562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:31.098783 11740 tablet_bootstrap.cc:492] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:31.099354 11742 tablet_bootstrap.cc:492] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:31.100911 11740 tablet_bootstrap.cc:654] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:31.101356 11742 tablet_bootstrap.cc:654] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:31.104315 11745 tablet_bootstrap.cc:492] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:31.104693 11740 tablet_bootstrap.cc:492] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:31.104871 11740 ts_tablet_manager.cc:1367] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:13:31.105777 11740 raft_consensus.cc:348] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.106015 11740 raft_consensus.cc:374] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:31.106158 11740 raft_consensus.cc:729] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.106428 11745 tablet_bootstrap.cc:654] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:31.106462 11740 consensus_queue.cc:260] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.108139 11740 ts_tablet_manager.cc:1398] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:31.108626 11740 tablet_bootstrap.cc:492] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:31.110975 11740 tablet_bootstrap.cc:654] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:31.111981 11742 tablet_bootstrap.cc:492] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:31.112322 11742 ts_tablet_manager.cc:1367] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20240503 08:13:31.115290 11741 tablet_bootstrap.cc:492] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:31.116040 11742 raft_consensus.cc:348] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:31.116365 11742 raft_consensus.cc:374] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:31.116530 11742 raft_consensus.cc:729] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.116595 11745 tablet_bootstrap.cc:492] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:31.116796 11740 tablet_bootstrap.cc:492] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:31.116823 11745 ts_tablet_manager.cc:1367] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20240503 08:13:31.117017 11740 ts_tablet_manager.cc:1367] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 08:13:31.117233 11741 tablet_bootstrap.cc:654] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:31.117834 11740 raft_consensus.cc:348] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.117866 11745 raft_consensus.cc:348] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.118084 11745 raft_consensus.cc:374] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:31.118129 11740 raft_consensus.cc:374] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:31.118254 11740 raft_consensus.cc:729] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.118268 11745 raft_consensus.cc:729] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.118325 11742 consensus_queue.cc:260] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:31.118564 11740 consensus_queue.cc:260] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.119259 11742 ts_tablet_manager.cc:1398] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:13:31.119254 11740 ts_tablet_manager.cc:1398] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:31.119838 11740 tablet_bootstrap.cc:492] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:31.118659 11745 consensus_queue.cc:260] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.120767 11745 ts_tablet_manager.cc:1398] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:13:31.121764 11740 tablet_bootstrap.cc:654] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:31.129613 11741 tablet_bootstrap.cc:492] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:31.129817 11741 ts_tablet_manager.cc:1367] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240503 08:13:31.130573 11741 raft_consensus.cc:348] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.130841 11741 raft_consensus.cc:374] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:31.130946 11741 raft_consensus.cc:729] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.131356 11741 consensus_queue.cc:260] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.132489 11741 ts_tablet_manager.cc:1398] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 08:13:31.134968 11741 tablet_bootstrap.cc:492] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:31.137334 11740 tablet_bootstrap.cc:492] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:31.138007 11740 ts_tablet_manager.cc:1367] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.012s
I20240503 08:13:31.140049 11741 tablet_bootstrap.cc:654] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:31.140648 11740 raft_consensus.cc:348] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:31.140784 11742 tablet_bootstrap.cc:492] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:31.140889 11740 raft_consensus.cc:374] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:31.140982 11740 raft_consensus.cc:729] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.141283 11740 consensus_queue.cc:260] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:31.142060 11740 ts_tablet_manager.cc:1398] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 08:13:31.145009 11742 tablet_bootstrap.cc:654] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:31.150960 11741 tablet_bootstrap.cc:492] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:31.150975 11742 tablet_bootstrap.cc:492] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:31.151188 11742 ts_tablet_manager.cc:1367] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.001s
I20240503 08:13:31.151178 11741 ts_tablet_manager.cc:1367] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20240503 08:13:31.152200 11741 raft_consensus.cc:348] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:31.152560 11741 raft_consensus.cc:374] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:31.152240 11742 raft_consensus.cc:348] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.152715 11741 raft_consensus.cc:729] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.152813 11742 raft_consensus.cc:374] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:31.152896 11742 raft_consensus.cc:729] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.153218 11741 consensus_queue.cc:260] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:31.153889 11741 ts_tablet_manager.cc:1398] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:31.154291 11741 tablet_bootstrap.cc:492] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:31.155356 11742 consensus_queue.cc:260] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.156143 11742 ts_tablet_manager.cc:1398] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:13:31.156594 11741 tablet_bootstrap.cc:654] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:31.160878 11741 tablet_bootstrap.cc:492] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:31.161037 11741 ts_tablet_manager.cc:1367] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 08:13:31.161707 11741 raft_consensus.cc:348] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.161881 11741 raft_consensus.cc:374] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:31.161975 11741 raft_consensus.cc:729] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:31.162195 11741 consensus_queue.cc:260] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.162708 11741 ts_tablet_manager.cc:1398] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:31.194310 11590 raft_consensus.cc:482] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:31.194497 11590 raft_consensus.cc:504] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.195380 11590 leader_election.cc:288] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:31.195894 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:31.195879 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:31.196122 10571 raft_consensus.cc:2454] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 0.
I20240503 08:13:31.196143 10308 raft_consensus.cc:2454] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 0.
I20240503 08:13:31.196574 10376 leader_election.cc:302] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:31.196856 11590 raft_consensus.cc:2781] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:31.197078 11590 raft_consensus.cc:482] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:31.197212 11590 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:31.199234 11590 raft_consensus.cc:504] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.200037 11590 leader_election.cc:288] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:31.200639 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:31.200639 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:31.200878 10571 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:31.200902 10308 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:31.203920 10308 raft_consensus.cc:2454] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 1.
I20240503 08:13:31.203928 10571 raft_consensus.cc:2454] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 1.
I20240503 08:13:31.204527 10374 leader_election.cc:302] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:31.204913 11590 raft_consensus.cc:2781] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:31.205067 11590 raft_consensus.cc:686] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:31.205379 11590 consensus_queue.cc:237] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.208357 10165 catalog_manager.cc:5484] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 1 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:31.223917 11703 raft_consensus.cc:482] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:31.224143 11703 raft_consensus.cc:504] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.224853 11703 leader_election.cc:288] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:31.225450 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:31.225517 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:31.225715 10571 raft_consensus.cc:2454] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:31.225746 10438 raft_consensus.cc:2454] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:31.226166 10637 leader_election.cc:302] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:31.226475 11703 raft_consensus.cc:2781] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:31.226595 11703 raft_consensus.cc:482] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:31.226686 11703 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:31.228698 11703 raft_consensus.cc:504] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.229398 11703 leader_election.cc:288] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:31.230074 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:31.230085 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:31.230334 10571 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:31.230311 10438 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:31.233350 10571 raft_consensus.cc:2454] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:31.233379 10438 raft_consensus.cc:2454] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:31.234026 10637 leader_election.cc:302] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:31.234385 11703 raft_consensus.cc:2781] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:31.234566 11703 raft_consensus.cc:686] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:31.234864 11703 consensus_queue.cc:237] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:31.238219 10165 catalog_manager.cc:5484] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:31.436815 11544 raft_consensus.cc:482] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:31.437016 11544 raft_consensus.cc:504] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:31.437779 11544 leader_election.cc:288] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:31.438480 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:31.438758 10699 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 0.
I20240503 08:13:31.439381 10504 leader_election.cc:302] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:31.439718 11544 raft_consensus.cc:2781] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:31.439847 11544 raft_consensus.cc:482] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:31.439972 11544 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:31.442747 11544 raft_consensus.cc:504] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:31.443599 11544 leader_election.cc:288] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:31.444110 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:31.444339 10699 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:31.445523 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:31.445822 10438 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 0.
I20240503 08:13:31.445755 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:31.446025 10440 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:31.446760 10699 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 1.
I20240503 08:13:31.447181 10504 leader_election.cc:302] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:31.447495 11544 raft_consensus.cc:2781] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:31.447925 11544 raft_consensus.cc:686] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:31.448261 11544 consensus_queue.cc:237] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:31.451162 10165 catalog_manager.cc:5484] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 1 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:31.481735 10440 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 1.
I20240503 08:13:31.636281 11590 consensus_queue.cc:1035] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:31.644764 11590 consensus_queue.cc:1035] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:31.715530 11703 consensus_queue.cc:1035] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:31.723809 11703 consensus_queue.cc:1035] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:31.861197 11759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:31.861554 11759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:31.870235 11759 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 08:13:31.979280 11544 consensus_queue.cc:1035] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:31.988394 11757 consensus_queue.cc:1035] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 256.295 ms
time per row: 0.128147 ms
I20240503 08:13:32.283607 10160 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44620:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:32.294387 10287 tablet_service.cc:1467] Processing CreateTablet for tablet 471f8f96b28a4af2b70604a7fd1628fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=5670ed458383412a90fea1953b0768e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:32.294775 10680 tablet_service.cc:1467] Processing CreateTablet for tablet 471f8f96b28a4af2b70604a7fd1628fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=5670ed458383412a90fea1953b0768e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:32.294974 10287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471f8f96b28a4af2b70604a7fd1628fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:32.295122 10419 tablet_service.cc:1467] Processing CreateTablet for tablet 471f8f96b28a4af2b70604a7fd1628fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=5670ed458383412a90fea1953b0768e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:32.295256 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471f8f96b28a4af2b70604a7fd1628fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:32.295604 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 471f8f96b28a4af2b70604a7fd1628fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:32.296221 10288 tablet_service.cc:1467] Processing CreateTablet for tablet d9595f5c7afc4218a96ef7a18a8d2e68 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5670ed458383412a90fea1953b0768e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:32.296666 10288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9595f5c7afc4218a96ef7a18a8d2e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:32.301559 10550 tablet_service.cc:1467] Processing CreateTablet for tablet d9595f5c7afc4218a96ef7a18a8d2e68 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5670ed458383412a90fea1953b0768e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:32.301975 10679 tablet_service.cc:1467] Processing CreateTablet for tablet d9595f5c7afc4218a96ef7a18a8d2e68 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5670ed458383412a90fea1953b0768e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:32.302134 10550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9595f5c7afc4218a96ef7a18a8d2e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:32.302417 10679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9595f5c7afc4218a96ef7a18a8d2e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:32.308929 10289 tablet_service.cc:1467] Processing CreateTablet for tablet 965d21fd6ae94abf96f96e224ccb7bdd (DEFAULT_TABLE table=rebalance_test_table_6 [id=5670ed458383412a90fea1953b0768e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:32.309401 10289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965d21fd6ae94abf96f96e224ccb7bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:32.312708 11826 tablet_bootstrap.cc:492] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:32.313095 10420 tablet_service.cc:1467] Processing CreateTablet for tablet 965d21fd6ae94abf96f96e224ccb7bdd (DEFAULT_TABLE table=rebalance_test_table_6 [id=5670ed458383412a90fea1953b0768e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:32.316061 11826 tablet_bootstrap.cc:654] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:32.316006 10679 tablet_service.cc:1467] Processing CreateTablet for tablet 965d21fd6ae94abf96f96e224ccb7bdd (DEFAULT_TABLE table=rebalance_test_table_6 [id=5670ed458383412a90fea1953b0768e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:32.316483 10679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965d21fd6ae94abf96f96e224ccb7bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:32.317229 11825 tablet_bootstrap.cc:492] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:32.320650 11825 tablet_bootstrap.cc:654] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:32.321008 10420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965d21fd6ae94abf96f96e224ccb7bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:32.333549 11824 tablet_bootstrap.cc:492] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:32.339030 11824 tablet_bootstrap.cc:654] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:32.345377 11828 tablet_bootstrap.cc:492] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:32.347577 11828 tablet_bootstrap.cc:654] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:32.350899 11825 tablet_bootstrap.cc:492] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:32.351054 11824 tablet_bootstrap.cc:492] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:32.351106 11825 ts_tablet_manager.cc:1367] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.019s
I20240503 08:13:32.351253 11824 ts_tablet_manager.cc:1367] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20240503 08:13:32.352109 11825 raft_consensus.cc:348] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.352371 11826 tablet_bootstrap.cc:492] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:32.352385 11825 raft_consensus.cc:374] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:32.352337 11824 raft_consensus.cc:348] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:32.352488 11825 raft_consensus.cc:729] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:32.352563 11824 raft_consensus.cc:374] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:32.352674 11824 raft_consensus.cc:729] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:32.352804 11826 ts_tablet_manager.cc:1367] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.010s
I20240503 08:13:32.352828 11825 consensus_queue.cc:260] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.352931 11824 consensus_queue.cc:260] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:32.353552 11824 ts_tablet_manager.cc:1398] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:32.353853 11824 tablet_bootstrap.cc:492] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:32.354039 11828 tablet_bootstrap.cc:492] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:32.354252 11828 ts_tablet_manager.cc:1367] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240503 08:13:32.355677 11824 tablet_bootstrap.cc:654] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:32.355705 11825 ts_tablet_manager.cc:1398] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 08:13:32.356042 11825 tablet_bootstrap.cc:492] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:32.357141 11826 raft_consensus.cc:348] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:32.357364 11826 raft_consensus.cc:374] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:32.357465 11826 raft_consensus.cc:729] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:32.355165 11828 raft_consensus.cc:348] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.358124 11828 raft_consensus.cc:374] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:32.358269 11828 raft_consensus.cc:729] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:32.358561 11828 consensus_queue.cc:260] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.359040 11825 tablet_bootstrap.cc:654] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:32.359983 11826 consensus_queue.cc:260] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:32.361032 11826 ts_tablet_manager.cc:1398] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 08:13:32.364504 11825 tablet_bootstrap.cc:492] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:32.364706 11825 ts_tablet_manager.cc:1367] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 08:13:32.365453 11825 raft_consensus.cc:348] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:32.365684 11825 raft_consensus.cc:374] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:32.365758 11825 raft_consensus.cc:729] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:32.366056 11825 consensus_queue.cc:260] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:32.368378 11824 tablet_bootstrap.cc:492] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:32.368557 11824 ts_tablet_manager.cc:1367] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20240503 08:13:32.368687 11828 ts_tablet_manager.cc:1398] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240503 08:13:32.369076 11828 tablet_bootstrap.cc:492] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:32.369376 11824 raft_consensus.cc:348] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.369593 11824 raft_consensus.cc:374] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:32.369663 11824 raft_consensus.cc:729] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:32.369979 11824 consensus_queue.cc:260] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.370522 11824 ts_tablet_manager.cc:1398] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 08:13:32.370862 11824 tablet_bootstrap.cc:492] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:32.371248 11828 tablet_bootstrap.cc:654] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:32.372107 11825 ts_tablet_manager.cc:1398] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 08:13:32.372877 11824 tablet_bootstrap.cc:654] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:32.375036 11443 raft_consensus.cc:482] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:32.375232 11443 raft_consensus.cc:504] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:32.375489 11825 tablet_bootstrap.cc:492] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:32.375982 11443 leader_election.cc:288] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:32.376725 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:32.376974 10570 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:32.377269 11825 tablet_bootstrap.cc:654] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:32.377830 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:32.378473 10245 leader_election.cc:302] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:32.378625 11824 tablet_bootstrap.cc:492] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:32.378798 11824 ts_tablet_manager.cc:1367] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 08:13:32.379035 10699 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:32.379753 11828 tablet_bootstrap.cc:492] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:32.379760 11824 raft_consensus.cc:348] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.379925 11828 ts_tablet_manager.cc:1367] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240503 08:13:32.380002 11824 raft_consensus.cc:374] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:32.380097 11824 raft_consensus.cc:729] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:32.380383 11824 consensus_queue.cc:260] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.382287 11443 raft_consensus.cc:2781] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:32.382448 11443 raft_consensus.cc:482] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:32.382565 11443 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:32.380842 11828 raft_consensus.cc:348] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.383730 11828 raft_consensus.cc:374] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:32.383841 11828 raft_consensus.cc:729] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:32.384207 11828 consensus_queue.cc:260] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.385075 11828 ts_tablet_manager.cc:1398] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 08:13:32.386149 11443 raft_consensus.cc:504] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:32.386694 11825 tablet_bootstrap.cc:492] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:32.386859 11825 ts_tablet_manager.cc:1367] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240503 08:13:32.387581 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:32.387780 11824 ts_tablet_manager.cc:1398] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 08:13:32.387812 10699 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:32.388594 11443 leader_election.cc:288] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:32.389400 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:32.389628 10571 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:32.391525 10699 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:32.387815 11825 raft_consensus.cc:348] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.392069 11825 raft_consensus.cc:374] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:32.392191 11825 raft_consensus.cc:729] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:32.392462 11825 consensus_queue.cc:260] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.392580 10571 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:32.392796 10242 leader_election.cc:302] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:32.393303 11838 raft_consensus.cc:2781] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:32.393517 11838 raft_consensus.cc:686] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:32.393718 11825 ts_tablet_manager.cc:1398] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 08:13:32.393998 11838 consensus_queue.cc:237] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:32.397894 10161 catalog_manager.cc:5484] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 1 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:32.467479 11578 raft_consensus.cc:482] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:32.467741 11578 raft_consensus.cc:504] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.468430 11578 leader_election.cc:288] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:32.469167 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:32.469202 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:32.469466 10308 raft_consensus.cc:2454] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:32.469507 10440 raft_consensus.cc:2454] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:32.470014 10635 leader_election.cc:302] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:32.470331 11578 raft_consensus.cc:2781] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:32.470471 11578 raft_consensus.cc:482] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:32.470587 11578 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:32.472623 11578 raft_consensus.cc:504] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.473253 11578 leader_election.cc:288] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:32.473932 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:32.473912 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:32.474149 10440 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:32.474149 10308 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:32.477135 10308 raft_consensus.cc:2454] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:32.477135 10440 raft_consensus.cc:2454] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:32.477723 10635 leader_election.cc:302] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:32.478085 11578 raft_consensus.cc:2781] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:32.478278 11578 raft_consensus.cc:686] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:32.478611 11578 consensus_queue.cc:237] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.481580 10161 catalog_manager.cc:5484] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:32.613445 11838 raft_consensus.cc:482] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:32.613659 11838 raft_consensus.cc:504] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.614310 11838 leader_election.cc:288] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:32.614980 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:32.615015 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:32.615285 10699 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:32.615280 10440 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:32.615866 10242 leader_election.cc:302] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:32.616199 11838 raft_consensus.cc:2781] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:32.616291 11838 raft_consensus.cc:482] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:32.616364 11838 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:32.618305 11838 raft_consensus.cc:504] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.618954 11838 leader_election.cc:288] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:32.619522 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:32.619526 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:32.619741 10699 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:32.619757 10440 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:32.621699 10440 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:32.621731 10699 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:32.622222 10245 leader_election.cc:302] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:32.622628 11838 raft_consensus.cc:2781] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:32.622808 11838 raft_consensus.cc:686] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:32.623091 11838 consensus_queue.cc:237] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:32.626353 10161 catalog_manager.cc:5484] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 1 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:32.829052 11838 consensus_queue.cc:1035] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:32.847142 11838 consensus_queue.cc:1035] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:32.893260 11578 consensus_queue.cc:1035] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:32.898640 11814 consensus_queue.cc:1035] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:33.024333 11843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:33.024698 11843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:33.032737 11843 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 08:13:33.070551 11838 consensus_queue.cc:1035] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:33.075908 11443 consensus_queue.cc:1035] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 288.358 ms
time per row: 0.144179 ms
I20240503 08:13:33.481459 10162 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44620:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:33.490746 10419 tablet_service.cc:1467] Processing CreateTablet for tablet f1dff69d2bf84a45bdd1cd443d833f30 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d873faba658846138a514475c8c2e48c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:33.490998 10550 tablet_service.cc:1467] Processing CreateTablet for tablet f1dff69d2bf84a45bdd1cd443d833f30 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d873faba658846138a514475c8c2e48c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:33.491462 10550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1dff69d2bf84a45bdd1cd443d833f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:33.491292 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1dff69d2bf84a45bdd1cd443d833f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:33.492081 10289 tablet_service.cc:1467] Processing CreateTablet for tablet d44e068b52864dde9d988775249cb8d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d873faba658846138a514475c8c2e48c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:33.492467 10289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44e068b52864dde9d988775249cb8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:33.492956 10549 tablet_service.cc:1467] Processing CreateTablet for tablet d44e068b52864dde9d988775249cb8d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d873faba658846138a514475c8c2e48c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:33.493355 10549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44e068b52864dde9d988775249cb8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:33.498253 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 4670994e75a24234913724bac4c30ec4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d873faba658846138a514475c8c2e48c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:33.498822 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4670994e75a24234913724bac4c30ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:33.499437 10679 tablet_service.cc:1467] Processing CreateTablet for tablet d44e068b52864dde9d988775249cb8d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d873faba658846138a514475c8c2e48c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:33.499867 10679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44e068b52864dde9d988775249cb8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:33.503791 10680 tablet_service.cc:1467] Processing CreateTablet for tablet 4670994e75a24234913724bac4c30ec4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d873faba658846138a514475c8c2e48c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:33.504360 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4670994e75a24234913724bac4c30ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:33.491258 10288 tablet_service.cc:1467] Processing CreateTablet for tablet f1dff69d2bf84a45bdd1cd443d833f30 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d873faba658846138a514475c8c2e48c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:33.508507 10288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1dff69d2bf84a45bdd1cd443d833f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:33.508749 10420 tablet_service.cc:1467] Processing CreateTablet for tablet 4670994e75a24234913724bac4c30ec4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d873faba658846138a514475c8c2e48c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:33.509347 10420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4670994e75a24234913724bac4c30ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:33.512612 11906 tablet_bootstrap.cc:492] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:33.517526 11906 tablet_bootstrap.cc:654] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:33.517614 11907 tablet_bootstrap.cc:492] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:33.519284 11907 tablet_bootstrap.cc:654] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:33.520457 11908 tablet_bootstrap.cc:492] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:33.521293 11906 tablet_bootstrap.cc:492] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:33.521543 11906 ts_tablet_manager.cc:1367] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 08:13:33.522791 11908 tablet_bootstrap.cc:654] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:33.526688 11906 raft_consensus.cc:348] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:33.527132 11906 raft_consensus.cc:374] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:33.527297 11906 raft_consensus.cc:729] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:33.527698 11906 consensus_queue.cc:260] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:33.529450 11906 ts_tablet_manager.cc:1398] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 08:13:33.529845 11906 tablet_bootstrap.cc:492] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:33.533567 11906 tablet_bootstrap.cc:654] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:33.535558 11910 tablet_bootstrap.cc:492] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:33.537413 11907 tablet_bootstrap.cc:492] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:33.537616 11907 ts_tablet_manager.cc:1367] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20240503 08:13:33.538154 11910 tablet_bootstrap.cc:654] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:33.538565 11907 raft_consensus.cc:348] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.538805 11907 raft_consensus.cc:374] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:33.538908 11907 raft_consensus.cc:729] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:33.539747 11908 tablet_bootstrap.cc:492] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:33.539970 11908 ts_tablet_manager.cc:1367] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.007s
I20240503 08:13:33.539228 11907 consensus_queue.cc:260] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.541859 11908 raft_consensus.cc:348] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:33.542184 11908 raft_consensus.cc:374] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:33.542311 11908 raft_consensus.cc:729] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:33.542644 11908 consensus_queue.cc:260] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:33.544054 11908 ts_tablet_manager.cc:1398] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:33.544441 11908 tablet_bootstrap.cc:492] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:33.547133 11908 tablet_bootstrap.cc:654] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:33.548314 11907 ts_tablet_manager.cc:1398] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 08:13:33.549469 11910 tablet_bootstrap.cc:492] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:33.549710 11910 ts_tablet_manager.cc:1367] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240503 08:13:33.550699 11910 raft_consensus.cc:348] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.551000 11910 raft_consensus.cc:374] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:33.551175 11910 raft_consensus.cc:729] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:33.551577 11910 consensus_queue.cc:260] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.552676 11910 ts_tablet_manager.cc:1398] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:33.553126 11907 tablet_bootstrap.cc:492] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:33.553289 11910 tablet_bootstrap.cc:492] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:33.554965 11907 tablet_bootstrap.cc:654] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:33.554963 11910 tablet_bootstrap.cc:654] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:33.560725 11910 tablet_bootstrap.cc:492] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:33.560791 11907 tablet_bootstrap.cc:492] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:33.560891 11906 tablet_bootstrap.cc:492] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:33.560938 11910 ts_tablet_manager.cc:1367] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 08:13:33.560964 11907 ts_tablet_manager.cc:1367] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240503 08:13:33.561079 11906 ts_tablet_manager.cc:1367] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.004s
I20240503 08:13:33.561707 11908 tablet_bootstrap.cc:492] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:33.561787 11910 raft_consensus.cc:348] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.561784 11906 raft_consensus.cc:348] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.562038 11910 raft_consensus.cc:374] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:33.562043 11906 raft_consensus.cc:374] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:33.561962 11908 ts_tablet_manager.cc:1367] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.001s
I20240503 08:13:33.562124 11906 raft_consensus.cc:729] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:33.562124 11910 raft_consensus.cc:729] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:33.562362 11906 consensus_queue.cc:260] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.562394 11910 consensus_queue.cc:260] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.562978 11910 ts_tablet_manager.cc:1398] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:33.563222 11907 raft_consensus.cc:348] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:33.563763 11906 ts_tablet_manager.cc:1398] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:33.564078 11906 tablet_bootstrap.cc:492] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:33.564075 11907 raft_consensus.cc:374] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:33.564002 11908 raft_consensus.cc:348] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.564160 11907 raft_consensus.cc:729] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:33.564267 11908 raft_consensus.cc:374] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:33.564345 11908 raft_consensus.cc:729] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:33.564599 11908 consensus_queue.cc:260] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.565953 11908 ts_tablet_manager.cc:1398] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 08:13:33.565145 11907 consensus_queue.cc:260] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:33.566468 11906 tablet_bootstrap.cc:654] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:33.569116 11907 ts_tablet_manager.cc:1398] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 08:13:33.569936 11906 tablet_bootstrap.cc:492] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:33.570101 11906 ts_tablet_manager.cc:1367] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 08:13:33.571125 11906 raft_consensus.cc:348] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.571530 11906 raft_consensus.cc:374] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:33.571632 11906 raft_consensus.cc:729] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:33.571851 11906 consensus_queue.cc:260] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.572387 11906 ts_tablet_manager.cc:1398] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:13:33.589716 11578 raft_consensus.cc:482] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:33.589994 11578 raft_consensus.cc:504] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.590850 11578 leader_election.cc:288] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:33.591521 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:33.591549 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:33.591830 10571 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:33.591856 10440 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:33.592331 10637 leader_election.cc:302] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:33.592648 11578 raft_consensus.cc:2781] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:33.592759 11578 raft_consensus.cc:482] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:33.592867 11578 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:33.595149 11578 raft_consensus.cc:504] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.595893 11578 leader_election.cc:288] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:33.596356 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:33.596385 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:33.596551 10440 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:33.596614 10570 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:33.598518 10440 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:33.598626 10570 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:33.599076 10637 leader_election.cc:302] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:33.599397 11578 raft_consensus.cc:2781] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:33.599587 11578 raft_consensus.cc:686] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:33.599861 11578 consensus_queue.cc:237] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.603137 10165 catalog_manager.cc:5484] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:33.605984 11839 raft_consensus.cc:482] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:33.606170 11839 raft_consensus.cc:504] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:33.606854 11839 leader_election.cc:288] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:33.607391 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:33.607477 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:33.607632 10440 raft_consensus.cc:2454] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:33.607749 10570 raft_consensus.cc:2454] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:33.608068 10245 leader_election.cc:302] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:33.608373 11839 raft_consensus.cc:2781] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:33.608498 11839 raft_consensus.cc:482] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:33.608590 11839 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:33.610838 11839 raft_consensus.cc:504] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:33.611603 11839 leader_election.cc:288] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:33.612092 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:33.612118 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:33.612298 10440 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:33.612339 10570 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:33.614362 10440 raft_consensus.cc:2454] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:33.614384 10570 raft_consensus.cc:2454] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:33.614892 10245 leader_election.cc:302] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:33.615278 11839 raft_consensus.cc:2781] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:33.615506 11839 raft_consensus.cc:686] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:33.615797 11839 consensus_queue.cc:237] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:33.619148 10165 catalog_manager.cc:5484] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 1 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:33.678169 11883 raft_consensus.cc:482] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:33.678381 11883 raft_consensus.cc:504] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.679194 11883 leader_election.cc:288] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:33.679895 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:33.679952 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:33.680177 10701 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 0.
I20240503 08:13:33.680186 10307 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 0.
I20240503 08:13:33.680697 10505 leader_election.cc:302] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:33.681165 11883 raft_consensus.cc:2781] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:33.681273 11883 raft_consensus.cc:482] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:33.681387 11883 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:33.684305 11883 raft_consensus.cc:504] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.684979 11883 leader_election.cc:288] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:33.685525 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:33.685561 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:33.685734 10307 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:33.685756 10701 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:33.687951 10701 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 1.
I20240503 08:13:33.688515 10504 leader_election.cc:302] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:33.688701 10307 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 1.
I20240503 08:13:33.689272 11883 raft_consensus.cc:2781] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:33.689852 11883 raft_consensus.cc:686] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:33.690156 11883 consensus_queue.cc:237] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:33.693722 10167 catalog_manager.cc:5484] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 1 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:33.994372 11839 consensus_queue.cc:1035] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:34.004907 11443 consensus_queue.cc:1035] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:34.012333 11578 consensus_queue.cc:1035] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:34.023893 11814 consensus_queue.cc:1035] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:34.110098 11883 consensus_queue.cc:1035] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:34.110003 11922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:34.110472 11922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:13:34.115743 11936 consensus_queue.cc:1035] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:34.122272 11922 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
INSERT report
rows total: 2000
time total: 387.477 ms
time per row: 0.193739 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 08:13:35.636274 10167 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44620:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:35.647317 10551 tablet_service.cc:1467] Processing CreateTablet for tablet f9f91eb635604d20853c9f7518164789 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f84d77e6744478eb3ba184f2f75100c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:35.647194 10288 tablet_service.cc:1467] Processing CreateTablet for tablet f9f91eb635604d20853c9f7518164789 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f84d77e6744478eb3ba184f2f75100c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:35.647687 10419 tablet_service.cc:1467] Processing CreateTablet for tablet f9f91eb635604d20853c9f7518164789 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f84d77e6744478eb3ba184f2f75100c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:35.647828 10288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f91eb635604d20853c9f7518164789. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.647831 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f91eb635604d20853c9f7518164789. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.648118 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f91eb635604d20853c9f7518164789. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.653980 10289 tablet_service.cc:1467] Processing CreateTablet for tablet 43fedbb54bc34fa49f69d4891323405b (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f84d77e6744478eb3ba184f2f75100c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:35.654436 10289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43fedbb54bc34fa49f69d4891323405b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.655115 10679 tablet_service.cc:1467] Processing CreateTablet for tablet 43fedbb54bc34fa49f69d4891323405b (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f84d77e6744478eb3ba184f2f75100c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:35.655715 10679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43fedbb54bc34fa49f69d4891323405b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.658978 11984 tablet_bootstrap.cc:492] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:35.661870 11984 tablet_bootstrap.cc:654] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:35.664973 10420 tablet_service.cc:1467] Processing CreateTablet for tablet 43fedbb54bc34fa49f69d4891323405b (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f84d77e6744478eb3ba184f2f75100c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:35.665275 10549 tablet_service.cc:1467] Processing CreateTablet for tablet 74a927d070284ea09cfc9884e2c4e2b7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f84d77e6744478eb3ba184f2f75100c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:35.665490 10420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43fedbb54bc34fa49f69d4891323405b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.665675 10549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a927d070284ea09cfc9884e2c4e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.667840 10680 tablet_service.cc:1467] Processing CreateTablet for tablet 74a927d070284ea09cfc9884e2c4e2b7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f84d77e6744478eb3ba184f2f75100c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:35.668329 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a927d070284ea09cfc9884e2c4e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.672642 10287 tablet_service.cc:1467] Processing CreateTablet for tablet 74a927d070284ea09cfc9884e2c4e2b7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f84d77e6744478eb3ba184f2f75100c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:35.673180 10287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a927d070284ea09cfc9884e2c4e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:35.675771 11988 tablet_bootstrap.cc:492] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:35.679960 11986 tablet_bootstrap.cc:492] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:35.681699 11986 tablet_bootstrap.cc:654] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:35.679293 11988 tablet_bootstrap.cc:654] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:35.688557 11988 tablet_bootstrap.cc:492] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:35.688822 11988 ts_tablet_manager.cc:1367] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.003s
I20240503 08:13:35.690423 11984 tablet_bootstrap.cc:492] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:35.690754 11984 ts_tablet_manager.cc:1367] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.010s
I20240503 08:13:35.691053 11988 raft_consensus.cc:348] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.691429 11988 raft_consensus.cc:374] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:35.691602 11988 raft_consensus.cc:729] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:35.691716 11984 raft_consensus.cc:348] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.691988 11984 raft_consensus.cc:374] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:35.691948 11988 consensus_queue.cc:260] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.692122 11984 raft_consensus.cc:729] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:35.692466 11984 consensus_queue.cc:260] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.692589 11988 ts_tablet_manager.cc:1398] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 08:13:35.694144 11984 ts_tablet_manager.cc:1398] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:35.694495 11984 tablet_bootstrap.cc:492] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:35.695159 11986 tablet_bootstrap.cc:492] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:35.695428 11986 ts_tablet_manager.cc:1367] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 08:13:35.696317 11984 tablet_bootstrap.cc:654] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:35.696336 11986 raft_consensus.cc:348] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.696573 11986 raft_consensus.cc:374] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:35.698662 11986 raft_consensus.cc:729] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:35.698973 11986 consensus_queue.cc:260] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.702543 11986 ts_tablet_manager.cc:1398] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 08:13:35.707839 11988 tablet_bootstrap.cc:492] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:35.704207 11986 tablet_bootstrap.cc:492] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:35.709146 11984 tablet_bootstrap.cc:492] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:35.709342 11984 ts_tablet_manager.cc:1367] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20240503 08:13:35.710215 11984 raft_consensus.cc:348] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.710456 11984 raft_consensus.cc:374] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:35.710537 11984 raft_consensus.cc:729] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:35.710826 11984 consensus_queue.cc:260] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.711233 11986 tablet_bootstrap.cc:654] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:35.712553 11984 ts_tablet_manager.cc:1398] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 08:13:35.713017 11988 tablet_bootstrap.cc:654] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:35.714850 11989 tablet_bootstrap.cc:492] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:35.716682 11989 tablet_bootstrap.cc:654] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:35.718819 11986 tablet_bootstrap.cc:492] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:35.719019 11986 ts_tablet_manager.cc:1367] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 08:13:35.719120 11988 tablet_bootstrap.cc:492] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:35.719308 11988 ts_tablet_manager.cc:1367] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20240503 08:13:35.719883 11986 raft_consensus.cc:348] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.720134 11986 raft_consensus.cc:374] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:35.720077 11988 raft_consensus.cc:348] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:35.720211 11986 raft_consensus.cc:729] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:35.720311 11988 raft_consensus.cc:374] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:35.720391 11988 raft_consensus.cc:729] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:35.720403 11989 tablet_bootstrap.cc:492] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:35.720492 11986 consensus_queue.cc:260] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.720702 11989 ts_tablet_manager.cc:1367] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 08:13:35.720631 11988 consensus_queue.cc:260] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:35.721056 11986 ts_tablet_manager.cc:1398] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:35.721168 11988 ts_tablet_manager.cc:1398] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 08:13:35.721743 11989 raft_consensus.cc:348] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:35.722151 11986 tablet_bootstrap.cc:492] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:35.722290 11989 raft_consensus.cc:374] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:35.722469 11989 raft_consensus.cc:729] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:35.722739 11989 consensus_queue.cc:260] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:35.723847 11989 ts_tablet_manager.cc:1398] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:35.724223 11989 tablet_bootstrap.cc:492] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:35.726403 11989 tablet_bootstrap.cc:654] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:35.726845 11986 tablet_bootstrap.cc:654] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:35.731565 11986 tablet_bootstrap.cc:492] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:35.731832 11986 ts_tablet_manager.cc:1367] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 08:13:35.732686 11986 raft_consensus.cc:348] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:35.732933 11986 raft_consensus.cc:374] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:35.733258 11986 raft_consensus.cc:729] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:35.733508 11986 consensus_queue.cc:260] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:35.734019 11986 ts_tablet_manager.cc:1398] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:35.734717 11989 tablet_bootstrap.cc:492] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:35.734910 11989 ts_tablet_manager.cc:1367] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 08:13:35.735859 11989 raft_consensus.cc:348] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.736135 11989 raft_consensus.cc:374] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:35.736231 11989 raft_consensus.cc:729] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:35.736498 11989 consensus_queue.cc:260] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:35.737046 11989 ts_tablet_manager.cc:1398] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:35.870322 11839 raft_consensus.cc:482] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:35.870549 11839 raft_consensus.cc:504] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:35.871306 11839 leader_election.cc:288] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:35.871932 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:35.872006 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:35.872220 10699 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:35.872234 10570 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:35.872813 10245 leader_election.cc:302] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:35.873171 11839 raft_consensus.cc:2781] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:35.873337 11839 raft_consensus.cc:482] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:35.873438 11839 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:35.875727 11839 raft_consensus.cc:504] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:35.876430 11839 leader_election.cc:288] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:35.876914 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:35.876942 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:35.877082 10570 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:35.877136 10699 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:35.879091 10570 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:35.879190 10699 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:35.879738 10242 leader_election.cc:302] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:35.880146 11839 raft_consensus.cc:2781] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:35.880573 11839 raft_consensus.cc:686] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:35.880951 11839 consensus_queue.cc:237] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:35.883972 10168 catalog_manager.cc:5484] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 1 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:36.003656 11578 raft_consensus.cc:482] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:36.003904 11578 raft_consensus.cc:504] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:36.004660 11578 leader_election.cc:288] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:36.005209 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:36.005295 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:36.005445 10308 raft_consensus.cc:2454] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:36.005555 10440 raft_consensus.cc:2454] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:36.005991 10635 leader_election.cc:302] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:36.006294 11578 raft_consensus.cc:2781] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:36.006384 11578 raft_consensus.cc:482] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:36.006469 11578 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:36.008512 11578 raft_consensus.cc:504] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:36.009147 11578 leader_election.cc:288] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:36.009739 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:36.009760 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:36.009992 10440 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:36.009996 10308 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:36.013036 10440 raft_consensus.cc:2454] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:36.013027 10308 raft_consensus.cc:2454] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:36.013633 10637 leader_election.cc:302] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:36.013964 11578 raft_consensus.cc:2781] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:36.014179 11578 raft_consensus.cc:686] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:36.014479 11578 consensus_queue.cc:237] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:36.017779 10168 catalog_manager.cc:5484] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:36.022617 12000 raft_consensus.cc:482] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:36.022787 12000 raft_consensus.cc:504] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:36.023561 12000 leader_election.cc:288] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:36.024077 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" is_pre_election: true
I20240503 08:13:36.024133 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:36.024343 10308 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 0.
I20240503 08:13:36.024369 10440 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 0.
I20240503 08:13:36.024753 10507 leader_election.cc:302] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:36.025131 12000 raft_consensus.cc:2781] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:36.025274 12000 raft_consensus.cc:482] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:36.025372 12000 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:36.027848 12000 raft_consensus.cc:504] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:36.028453 12000 leader_election.cc:288] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:36.028975 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:36.029186 10440 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:36.029176 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:36.029403 10308 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:36.031196 10440 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 1.
I20240503 08:13:36.031618 10308 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 1.
I20240503 08:13:36.031725 10507 leader_election.cc:302] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:36.032055 12000 raft_consensus.cc:2781] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:36.032186 12000 raft_consensus.cc:686] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:36.032595 12000 consensus_queue.cc:237] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:36.035912 10168 catalog_manager.cc:5484] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 1 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:36.408413 11839 consensus_queue.cc:1035] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:36.410473 11578 consensus_queue.cc:1035] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:36.429339 11578 consensus_queue.cc:1035] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:36.436601 12005 consensus_queue.cc:1035] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:36.498852 12000 consensus_queue.cc:1035] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:36.500176 12003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:36.500627 12003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 08:13:36.505347 12018 consensus_queue.cc:1035] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:36.512140 12003 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 339.81 ms
time per row: 0.169905 ms
I20240503 08:13:37.016525 10166 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44620:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 08:13:37.026891 10420 tablet_service.cc:1467] Processing CreateTablet for tablet 00782a28ed184a198567f3bd7808a5bb (DEFAULT_TABLE table=rebalance_test_table_9 [id=d4412fe41acd49e99b5a8d42a1a1b098]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.028127 10549 tablet_service.cc:1467] Processing CreateTablet for tablet 00782a28ed184a198567f3bd7808a5bb (DEFAULT_TABLE table=rebalance_test_table_9 [id=d4412fe41acd49e99b5a8d42a1a1b098]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.028278 10419 tablet_service.cc:1467] Processing CreateTablet for tablet 71e2083a71dc4e9eb48eaa379874328d (DEFAULT_TABLE table=rebalance_test_table_9 [id=d4412fe41acd49e99b5a8d42a1a1b098]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.028600 10549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00782a28ed184a198567f3bd7808a5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.028661 10680 tablet_service.cc:1467] Processing CreateTablet for tablet 00782a28ed184a198567f3bd7808a5bb (DEFAULT_TABLE table=rebalance_test_table_9 [id=d4412fe41acd49e99b5a8d42a1a1b098]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.028683 10679 tablet_service.cc:1467] Processing CreateTablet for tablet 71e2083a71dc4e9eb48eaa379874328d (DEFAULT_TABLE table=rebalance_test_table_9 [id=d4412fe41acd49e99b5a8d42a1a1b098]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.028926 10418 tablet_service.cc:1467] Processing CreateTablet for tablet 3c422532250a4ba3bf1553722462b9da (DEFAULT_TABLE table=rebalance_test_table_9 [id=d4412fe41acd49e99b5a8d42a1a1b098]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.029120 10679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e2083a71dc4e9eb48eaa379874328d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.029309 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00782a28ed184a198567f3bd7808a5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.029361 10420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00782a28ed184a198567f3bd7808a5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.031755 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 71e2083a71dc4e9eb48eaa379874328d (DEFAULT_TABLE table=rebalance_test_table_9 [id=d4412fe41acd49e99b5a8d42a1a1b098]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.032166 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e2083a71dc4e9eb48eaa379874328d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.033705 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c422532250a4ba3bf1553722462b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.035737 10287 tablet_service.cc:1467] Processing CreateTablet for tablet 3c422532250a4ba3bf1553722462b9da (DEFAULT_TABLE table=rebalance_test_table_9 [id=d4412fe41acd49e99b5a8d42a1a1b098]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.036198 10287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c422532250a4ba3bf1553722462b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.039469 12069 tablet_bootstrap.cc:492] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:37.040472 10550 tablet_service.cc:1467] Processing CreateTablet for tablet 3c422532250a4ba3bf1553722462b9da (DEFAULT_TABLE table=rebalance_test_table_9 [id=d4412fe41acd49e99b5a8d42a1a1b098]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 08:13:37.040902 10550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c422532250a4ba3bf1553722462b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.042312 12069 tablet_bootstrap.cc:654] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.042661 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e2083a71dc4e9eb48eaa379874328d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:13:37.048580 12069 tablet_bootstrap.cc:492] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:37.048760 12069 ts_tablet_manager.cc:1367] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240503 08:13:37.049607 12069 raft_consensus.cc:348] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:37.049836 12069 raft_consensus.cc:374] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.049922 12069 raft_consensus.cc:729] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.050194 12069 consensus_queue.cc:260] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:37.050907 12069 ts_tablet_manager.cc:1398] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 08:13:37.051458 12069 tablet_bootstrap.cc:492] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931: Bootstrap starting.
I20240503 08:13:37.061142 12069 tablet_bootstrap.cc:654] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.062386 12071 tablet_bootstrap.cc:492] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:37.062923 12072 tablet_bootstrap.cc:492] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:37.064628 12071 tablet_bootstrap.cc:654] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.066418 12073 tablet_bootstrap.cc:492] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a: Bootstrap starting.
I20240503 08:13:37.068013 12072 tablet_bootstrap.cc:654] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.068051 12073 tablet_bootstrap.cc:654] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.068475 12069 tablet_bootstrap.cc:492] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931: No bootstrap required, opened a new log
I20240503 08:13:37.068660 12069 ts_tablet_manager.cc:1367] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 08:13:37.069736 12071 tablet_bootstrap.cc:492] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:37.069607 12069 raft_consensus.cc:348] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.069888 12071 ts_tablet_manager.cc:1367] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 08:13:37.069931 12069 raft_consensus.cc:374] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.070034 12069 raft_consensus.cc:729] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.070354 12069 consensus_queue.cc:260] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.070590 12071 raft_consensus.cc:348] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.070852 12071 raft_consensus.cc:374] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.070948 12071 raft_consensus.cc:729] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.071180 12071 consensus_queue.cc:260] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.071841 12071 ts_tablet_manager.cc:1398] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:37.072144 12071 tablet_bootstrap.cc:492] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:37.072227 12069 ts_tablet_manager.cc:1398] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:37.073762 12071 tablet_bootstrap.cc:654] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.077621 12072 tablet_bootstrap.cc:492] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:37.077790 12072 ts_tablet_manager.cc:1367] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240503 08:13:37.078012 12073 tablet_bootstrap.cc:492] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a: No bootstrap required, opened a new log
I20240503 08:13:37.078197 12073 ts_tablet_manager.cc:1367] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240503 08:13:37.078642 12072 raft_consensus.cc:348] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.078814 12072 raft_consensus.cc:374] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.078923 12072 raft_consensus.cc:729] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.079130 12072 consensus_queue.cc:260] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.079071 12073 raft_consensus.cc:348] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.079283 12073 raft_consensus.cc:374] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.079456 12073 raft_consensus.cc:729] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.079790 12072 ts_tablet_manager.cc:1398] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:37.079715 12073 consensus_queue.cc:260] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.080248 12072 tablet_bootstrap.cc:492] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:37.080278 12073 ts_tablet_manager.cc:1398] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:37.081751 12071 tablet_bootstrap.cc:492] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:37.081950 12071 ts_tablet_manager.cc:1367] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20240503 08:13:37.082803 12071 raft_consensus.cc:348] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.083070 12071 raft_consensus.cc:374] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.083148 12071 raft_consensus.cc:729] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.083428 12071 consensus_queue.cc:260] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.084136 12071 ts_tablet_manager.cc:1398] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 08:13:37.084411 12071 tablet_bootstrap.cc:492] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2: Bootstrap starting.
I20240503 08:13:37.085888 12071 tablet_bootstrap.cc:654] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.087375 12072 tablet_bootstrap.cc:654] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.092021 12071 tablet_bootstrap.cc:492] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2: No bootstrap required, opened a new log
I20240503 08:13:37.092028 12072 tablet_bootstrap.cc:492] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:37.092196 12071 ts_tablet_manager.cc:1367] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 08:13:37.092203 12072 ts_tablet_manager.cc:1367] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20240503 08:13:37.092878 12071 raft_consensus.cc:348] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:37.092859 12072 raft_consensus.cc:348] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:37.093086 12071 raft_consensus.cc:374] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.093092 12072 raft_consensus.cc:374] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.093165 12072 raft_consensus.cc:729] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.093189 12071 raft_consensus.cc:729] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.093420 12072 consensus_queue.cc:260] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:37.093483 12071 consensus_queue.cc:260] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:37.094094 12071 ts_tablet_manager.cc:1398] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:37.094524 12072 ts_tablet_manager.cc:1398] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 08:13:37.095219 12072 tablet_bootstrap.cc:492] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71: Bootstrap starting.
I20240503 08:13:37.096555 12072 tablet_bootstrap.cc:654] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71: Neither blocks nor log segments found. Creating new log.
I20240503 08:13:37.098704 12072 tablet_bootstrap.cc:492] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71: No bootstrap required, opened a new log
I20240503 08:13:37.098858 12072 ts_tablet_manager.cc:1367] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 08:13:37.099730 12072 raft_consensus.cc:348] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.099926 12072 raft_consensus.cc:374] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:13:37.100021 12072 raft_consensus.cc:729] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Initialized, Role: FOLLOWER
I20240503 08:13:37.100258 12072 consensus_queue.cc:260] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.100737 12072 ts_tablet_manager.cc:1398] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:13:37.147853 11985 raft_consensus.cc:482] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:37.148087 11985 raft_consensus.cc:504] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.148850 11985 leader_election.cc:288] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:37.149432 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931" is_pre_election: true
I20240503 08:13:37.149425 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:37.149672 10571 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 0.
I20240503 08:13:37.149677 10701 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 0.
I20240503 08:13:37.150447 10373 leader_election.cc:302] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:37.151079 11985 raft_consensus.cc:2781] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:37.151293 11985 raft_consensus.cc:482] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:37.151499 11985 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.154016 11985 raft_consensus.cc:504] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.155170 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:37.155213 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:37.155401 10571 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.155503 10701 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.157402 10571 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 1.
I20240503 08:13:37.157657 11985 leader_election.cc:288] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:37.157704 10701 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 1.
I20240503 08:13:37.157871 10376 leader_election.cc:302] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:37.158197 11985 raft_consensus.cc:2781] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:37.158604 11985 raft_consensus.cc:686] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:37.158960 11985 consensus_queue.cc:237] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.162046 10161 catalog_manager.cc:5484] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 1 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:37.238788 11578 raft_consensus.cc:482] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:37.239024 11578 raft_consensus.cc:504] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:37.239663 11578 leader_election.cc:288] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:37.240175 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:37.240151 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:37.240414 10438 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:37.240392 10571 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 0.
I20240503 08:13:37.240839 10637 leader_election.cc:302] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:37.241091 11578 raft_consensus.cc:2781] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:37.241209 11578 raft_consensus.cc:482] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:37.241297 11578 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.243219 11578 raft_consensus.cc:504] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:37.243840 11578 leader_election.cc:288] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:37.244387 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:37.244427 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:37.244604 10571 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.244609 10438 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.246948 10438 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:37.246981 10571 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 1.
I20240503 08:13:37.247452 10637 leader_election.cc:302] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:37.247794 11578 raft_consensus.cc:2781] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:37.248059 11578 raft_consensus.cc:686] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:37.248315 11578 consensus_queue.cc:237] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:37.251677 10161 catalog_manager.cc:5484] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 1 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:37.320107 12005 raft_consensus.cc:482] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:13:37.320318 12005 raft_consensus.cc:504] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.320955 12005 leader_election.cc:288] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:37.321539 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2" is_pre_election: true
I20240503 08:13:37.321513 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71" is_pre_election: true
I20240503 08:13:37.321750 10438 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:37.321775 10571 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 0.
I20240503 08:13:37.322146 10245 leader_election.cc:302] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:37.322752 11839 raft_consensus.cc:2781] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:13:37.322852 11839 raft_consensus.cc:482] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:13:37.322959 11839 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.325139 11839 raft_consensus.cc:504] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.325686 11839 leader_election.cc:288] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:37.326192 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:37.326414 10438 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.326367 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:37.326606 10571 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:13:37.328714 10571 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:37.329264 10245 leader_election.cc:302] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:37.329303 10438 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 1.
I20240503 08:13:37.329630 12005 raft_consensus.cc:2781] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:13:37.329799 12005 raft_consensus.cc:686] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:37.330201 12005 consensus_queue.cc:237] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:37.333700 10161 catalog_manager.cc:5484] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 1 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:37.591236 11985 consensus_queue.cc:1035] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:37.599450 12082 consensus_queue.cc:1035] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:13:37.726475 12086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:37.726855 12086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:37.734789 12086 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 08:13:37.756719 11578 consensus_queue.cc:1035] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:37.762859 11578 consensus_queue.cc:1035] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:37.765800 12005 consensus_queue.cc:1035] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:13:37.776216 12110 consensus_queue.cc:1035] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 290.59 ms
time per row: 0.145295 ms
I20240503 08:13:38.190316 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.197:43391
--local_ip_for_outbound_sockets=127.27.157.197
--tserver_master_addrs=127.27.157.254:35531
--webserver_port=42783
--webserver_interface=127.27.157.197
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 08:13:38.559170 12150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:38.559633 12150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:38.559775 12150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:38.568099 12150 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:38.568264 12150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:38.568428 12150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.197
I20240503 08:13:38.578536 12150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.197:43391
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.157.197
--webserver_port=42783
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.197
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:38.580176 12150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:38.582094 12150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:38.596397 12156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:38.596748 12157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:38.597327 12159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:38.598268 12150 server_base.cc:1034] running on GCE node
I20240503 08:13:38.598959 12150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:38.600064 12150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:38.601347 12150 hybrid_clock.cc:648] HybridClock initialized: now 1714724018601252 us; error 56 us; skew 500 ppm
I20240503 08:13:38.601738 12150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:38.604429 12150 webserver.cc:460] Webserver started at http://127.27.157.197:42783/ using document root <none> and password file <none>
I20240503 08:13:38.605242 12150 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:38.605405 12150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:38.611615 12150 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 08:13:38.614892 12165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:38.616588 12150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:38.616806 12150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b327eb1caa0d40e58fc9dbb2409fa85e"
format_stamp: "Formatted at 2024-05-03 08:13:18 on dist-test-slave-dj8w"
I20240503 08:13:38.617750 12150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:38.640487 12150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:38.641423 12150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:38.641784 12150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:38.642902 12150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:38.645045 12150 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:38.645149 12150 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:38.645344 12150 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:38.645414 12150 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:38.702297 12150 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.197:43391
I20240503 08:13:38.702317 12277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.197:43391 every 8 connection(s)
I20240503 08:13:38.703876 12150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 08:13:38.705698 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 12150
I20240503 08:13:38.706331 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.198:40735
--local_ip_for_outbound_sockets=127.27.157.198
--tserver_master_addrs=127.27.157.254:35531
--webserver_port=33089
--webserver_interface=127.27.157.198
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:38.720571 12278 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:38.720999 12278 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:38.722062 12278 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:38.724035 10162 ts_manager.cc:194] Re-registered known tserver with Master: b327eb1caa0d40e58fc9dbb2409fa85e (127.27.157.197:43391)
I20240503 08:13:38.725435 10162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.197:41341
W20240503 08:13:39.081349 12282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:39.081722 12282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:39.081959 12282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:39.089877 12282 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:39.090025 12282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:39.090200 12282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.198
I20240503 08:13:39.100703 12282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.198:40735
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.157.198
--webserver_port=33089
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.198
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:39.102185 12282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:39.104336 12282 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:39.117449 12288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.117873 12289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.118608 12291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:39.119436 12282 server_base.cc:1034] running on GCE node
I20240503 08:13:39.120133 12282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:39.121191 12282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:39.122428 12282 hybrid_clock.cc:648] HybridClock initialized: now 1714724019122385 us; error 39 us; skew 500 ppm
I20240503 08:13:39.122833 12282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:39.125532 12282 webserver.cc:460] Webserver started at http://127.27.157.198:33089/ using document root <none> and password file <none>
I20240503 08:13:39.126305 12282 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:39.126435 12282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:39.132081 12282 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 08:13:39.135288 12297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.137197 12282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 08:13:39.137382 12282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "22887faab4ec4e3caa006fee5b91099b"
format_stamp: "Formatted at 2024-05-03 08:13:18 on dist-test-slave-dj8w"
I20240503 08:13:39.138515 12282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:39.172469 12282 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:39.173382 12282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:39.173682 12282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:39.174733 12282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:39.176357 12282 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:39.176445 12282 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.176582 12282 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:39.176640 12282 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.215135 12282 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.198:40735
I20240503 08:13:39.215193 12409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.198:40735 every 8 connection(s)
I20240503 08:13:39.216794 12282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 08:13:39.217258 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 12282
I20240503 08:13:39.218066 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.199:35937
--local_ip_for_outbound_sockets=127.27.157.199
--tserver_master_addrs=127.27.157.254:35531
--webserver_port=42845
--webserver_interface=127.27.157.199
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:39.233881 12410 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:39.234252 12410 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:39.235445 12410 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:39.237241 10162 ts_manager.cc:194] Re-registered known tserver with Master: 22887faab4ec4e3caa006fee5b91099b (127.27.157.198:40735)
I20240503 08:13:39.238320 10162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.198:49849
W20240503 08:13:39.590698 12413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:39.591050 12413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:39.591202 12413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:39.599588 12413 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:39.599747 12413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:39.599911 12413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.199
I20240503 08:13:39.609568 12413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.199:35937
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.157.199
--webserver_port=42845
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.199
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:39.611145 12413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:39.613299 12413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:39.627424 12420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.628947 12422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:39.629624 12419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:39.630260 12413 server_base.cc:1034] running on GCE node
I20240503 08:13:39.630903 12413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:39.632014 12413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:39.633214 12413 hybrid_clock.cc:648] HybridClock initialized: now 1714724019633153 us; error 54 us; skew 500 ppm
I20240503 08:13:39.633567 12413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:39.636175 12413 webserver.cc:460] Webserver started at http://127.27.157.199:42845/ using document root <none> and password file <none>
I20240503 08:13:39.637001 12413 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:39.637131 12413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:39.642989 12413 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 08:13:39.646353 12428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.647927 12413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 08:13:39.648108 12413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e14ef04b4a534fc1aa67ebfe5bdfceeb"
format_stamp: "Formatted at 2024-05-03 08:13:19 on dist-test-slave-dj8w"
I20240503 08:13:39.648978 12413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:39.666831 12413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:39.667834 12413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:39.668113 12413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:39.669220 12413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:39.671008 12413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:39.671082 12413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.671193 12413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:39.671237 12413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:39.708772 12413 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.199:35937
I20240503 08:13:39.708782 12540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.199:35937 every 8 connection(s)
I20240503 08:13:39.710577 12413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 08:13:39.717978 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 12413
I20240503 08:13:39.718677 28279 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpimNqO/build/asan/bin/kudu
/tmp/dist-test-taskpimNqO/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.157.200:45347
--local_ip_for_outbound_sockets=127.27.157.200
--tserver_master_addrs=127.27.157.254:35531
--webserver_port=37353
--webserver_interface=127.27.157.200
--builtin_ntp_servers=127.27.157.212:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 08:13:39.724748 12541 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:39.725198 12541 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:39.726224 12541 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:39.727869 10162 ts_manager.cc:194] Re-registered known tserver with Master: e14ef04b4a534fc1aa67ebfe5bdfceeb (127.27.157.199:35937)
I20240503 08:13:39.728276 12278 heartbeater.cc:503] Master 127.27.157.254:35531 was elected leader, sending a full tablet report...
I20240503 08:13:39.728873 10162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.199:59773
W20240503 08:13:40.085634 12544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 08:13:40.085942 12544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 08:13:40.086033 12544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 08:13:40.093972 12544 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 08:13:40.094117 12544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 08:13:40.094233 12544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.157.200
I20240503 08:13:40.104548 12544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.157.212:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.157.200:45347
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.157.200
--webserver_port=37353
--tserver_master_addrs=127.27.157.254:35531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.157.200
--log_dir=/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 07:43:23 UTC on efd1337a2aa3
build id 103
ASAN enabled
I20240503 08:13:40.105908 12544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:13:40.107810 12544 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:13:40.122696 12552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:40.123526 12551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:13:40.123898 12554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:13:40.124819 12544 server_base.cc:1034] running on GCE node
I20240503 08:13:40.125722 12544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 08:13:40.126999 12544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 08:13:40.128247 12544 hybrid_clock.cc:648] HybridClock initialized: now 1714724020128183 us; error 55 us; skew 500 ppm
I20240503 08:13:40.128616 12544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 08:13:40.131106 12544 webserver.cc:460] Webserver started at http://127.27.157.200:37353/ using document root <none> and password file <none>
I20240503 08:13:40.131929 12544 fs_manager.cc:359] Metadata directory not provided
I20240503 08:13:40.132066 12544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:13:40.138228 12544 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 08:13:40.141430 12560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:40.143195 12544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 08:13:40.143433 12544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4a4e71bb1f1c41afbaf4497af3ddd640"
format_stamp: "Formatted at 2024-05-03 08:13:19 on dist-test-slave-dj8w"
I20240503 08:13:40.144410 12544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:13:40.171573 12544 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:13:40.172492 12544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:13:40.172798 12544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:13:40.173833 12544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:13:40.175449 12544 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:13:40.175544 12544 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:40.175684 12544 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:13:40.175740 12544 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:13:40.207943 12544 rpc_server.cc:307] RPC server started. Bound to: 127.27.157.200:45347
I20240503 08:13:40.207988 12672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.157.200:45347 every 8 connection(s)
I20240503 08:13:40.209470 12544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpimNqO/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723865426541-28279-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 08:13:40.214953 28279 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpimNqO/build/asan/bin/kudu as pid 12544
I20240503 08:13:40.225862 12673 heartbeater.cc:347] Connected to a master server at 127.27.157.254:35531
I20240503 08:13:40.226300 12673 heartbeater.cc:464] Registering TS with master...
I20240503 08:13:40.228458 12673 heartbeater.cc:511] Master 127.27.157.254:35531 requested a full tablet report, sending...
I20240503 08:13:40.230132 10160 ts_manager.cc:194] Re-registered known tserver with Master: 4a4e71bb1f1c41afbaf4497af3ddd640 (127.27.157.200:45347)
I20240503 08:13:40.231015 10160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.157.200:55587
I20240503 08:13:40.241814 12410 heartbeater.cc:503] Master 127.27.157.254:35531 was elected leader, sending a full tablet report...
I20240503 08:13:40.272639 10699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.272958 10699 raft_consensus.cc:482] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.273069 10699 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.275717 10699 raft_consensus.cc:504] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.276639 10699 leader_election.cc:288] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.277460 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.277667 10569 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.279731 10569 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.280742 10637 leader_election.cc:302] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:40.281523 12414 raft_consensus.cc:2781] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.281991 12414 raft_consensus.cc:686] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.282367 12414 consensus_queue.cc:237] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.283767 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:40.284063 10438 raft_consensus.cc:3032] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.284183 10438 raft_consensus.cc:729] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.284638 10438 consensus_queue.cc:260] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.285353 10438 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.286721 10160 catalog_manager.cc:5484] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.288867 10438 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.290390 10699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.290630 10699 raft_consensus.cc:465] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.291764 10699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.291965 10699 raft_consensus.cc:482] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.292104 10699 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.294847 10699 raft_consensus.cc:504] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.296231 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.296483 10569 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.296429 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.296653 10308 raft_consensus.cc:3032] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.296732 10308 raft_consensus.cc:729] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.297075 10308 consensus_queue.cc:260] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.297590 10308 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.298647 10569 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.299229 10637 leader_election.cc:302] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:40.299731 12414 raft_consensus.cc:2781] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.299949 12414 raft_consensus.cc:686] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.300217 12414 consensus_queue.cc:237] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.300602 10308 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.301609 10699 leader_election.cc:288] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:40.302914 10699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4670994e75a24234913724bac4c30ec4"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.303229 10699 raft_consensus.cc:465] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.303836 10160 catalog_manager.cc:5484] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.312407 10699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.312909 10699 raft_consensus.cc:482] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.313201 10699 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.317184 10699 raft_consensus.cc:504] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.318110 10699 leader_election.cc:288] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.318497 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.318496 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.318678 10569 raft_consensus.cc:3032] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.318740 10308 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.318755 10569 raft_consensus.cc:729] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:40.319031 10569 consensus_queue.cc:260] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.319532 10569 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.322075 10569 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.322623 10637 leader_election.cc:302] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:40.323074 12414 raft_consensus.cc:2781] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.323215 12414 raft_consensus.cc:686] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.323371 10308 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.323573 12414 consensus_queue.cc:237] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.324453 10699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4764c2955533410188d40d9465156bcc"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.324733 10699 raft_consensus.cc:482] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.324939 10699 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.329058 10699 raft_consensus.cc:504] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.329869 10160 catalog_manager.cc:5484] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.330574 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.330785 10308 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.332469 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.332676 10569 raft_consensus.cc:3032] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.332768 10569 raft_consensus.cc:729] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:40.333097 10569 consensus_queue.cc:260] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.333505 10569 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.333658 10308 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.334187 10635 leader_election.cc:302] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.334522 12414 raft_consensus.cc:2781] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.334679 12414 raft_consensus.cc:686] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.334932 12414 consensus_queue.cc:237] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.336436 10569 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.336875 10699 leader_election.cc:288] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:40.338317 10699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54e6601a404c46248232c7193aacaecb"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.338551 10699 raft_consensus.cc:482] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.338692 10699 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.339699 10160 catalog_manager.cc:5484] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.341620 10699 raft_consensus.cc:504] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.342386 10699 leader_election.cc:288] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.343197 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:40.343478 10438 raft_consensus.cc:3032] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.343576 10438 raft_consensus.cc:729] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.343586 10701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.343758 10701 raft_consensus.cc:465] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.343891 10438 consensus_queue.cc:260] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.344358 10438 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.344820 10701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8127b76db73402d8100faad68271644"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.344976 10701 raft_consensus.cc:465] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.345888 10701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.346056 10701 raft_consensus.cc:465] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.346966 10701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab9203ab3c96465682075b911880b320"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.347136 10701 raft_consensus.cc:465] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.347175 10438 raft_consensus.cc:2454] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.347883 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.348089 10637 leader_election.cc:302] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:40.348130 10701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.348181 10569 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.348321 10701 raft_consensus.cc:465] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.348486 12414 raft_consensus.cc:2781] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.349073 12414 raft_consensus.cc:686] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.349380 10701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.349545 10701 raft_consensus.cc:482] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.349457 12414 consensus_queue.cc:237] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.349666 10701 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.351045 10569 raft_consensus.cc:2454] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.353183 10701 raft_consensus.cc:504] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.354586 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:40.354585 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.354888 10438 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.354861 10308 raft_consensus.cc:3032] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.354980 10308 raft_consensus.cc:729] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.354292 10701 leader_election.cc:288] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:40.355315 10308 consensus_queue.cc:260] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.355728 10308 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.357018 10438 raft_consensus.cc:2454] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.357637 10637 leader_election.cc:302] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:40.358026 12047 raft_consensus.cc:2781] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.358168 12047 raft_consensus.cc:686] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.358485 12047 consensus_queue.cc:237] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.360411 10701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3715b584822e4cd3bee05119936125d0"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.360690 10701 raft_consensus.cc:465] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.361879 10701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.362138 10701 raft_consensus.cc:482] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.362282 10701 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.365247 10701 raft_consensus.cc:504] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.361657 10308 raft_consensus.cc:2454] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.366263 10160 catalog_manager.cc:5484] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.366503 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.366721 10307 raft_consensus.cc:3032] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.366801 10307 raft_consensus.cc:729] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.367013 10701 leader_election.cc:288] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:40.367094 10307 consensus_queue.cc:260] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.367475 10307 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.368032 10699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "205f68b8cf6d4b948576765252160224"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.368201 10699 raft_consensus.cc:482] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.368309 10699 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.370160 10307 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.370827 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.370873 10699 raft_consensus.cc:504] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.371062 10569 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.371623 10699 leader_election.cc:288] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:40.372586 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:40.372798 10438 raft_consensus.cc:3032] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.372907 10438 raft_consensus.cc:729] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.373176 10438 consensus_queue.cc:260] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.373754 10438 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.373404 10160 catalog_manager.cc:5484] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.373832 10569 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.374291 10637 leader_election.cc:302] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:40.376096 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.376453 10307 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.378296 12414 raft_consensus.cc:2781] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.378315 10438 raft_consensus.cc:2454] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.379786 10637 leader_election.cc:302] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:40.379779 12414 raft_consensus.cc:686] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.380147 12047 raft_consensus.cc:2781] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.380311 12047 raft_consensus.cc:686] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.380374 12414 consensus_queue.cc:237] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.380548 12047 consensus_queue.cc:237] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.384799 10307 raft_consensus.cc:2454] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.385442 10699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.385856 10699 raft_consensus.cc:465] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.387131 10699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.387465 10699 raft_consensus.cc:482] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.387684 10699 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.389173 10160 catalog_manager.cc:5484] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.390813 10699 raft_consensus.cc:504] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.391577 10699 leader_election.cc:288] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.392639 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.392686 10701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d01df9a11104a33a7971e133310acce"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.392851 10569 raft_consensus.cc:3032] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.392907 10701 raft_consensus.cc:482] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.392963 10569 raft_consensus.cc:729] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:40.393041 10701 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.393219 10569 consensus_queue.cc:260] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.393512 10569 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.395565 10569 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.395920 10701 raft_consensus.cc:504] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.396071 10637 leader_election.cc:302] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:40.396494 12414 raft_consensus.cc:2781] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.396718 12414 raft_consensus.cc:686] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.397039 12414 consensus_queue.cc:237] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.397356 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:40.397639 10438 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.399541 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.399987 10569 raft_consensus.cc:3032] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.400146 10569 raft_consensus.cc:729] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:40.400539 10569 consensus_queue.cc:260] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.401082 10569 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.400863 10160 catalog_manager.cc:5484] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.403810 10701 leader_election.cc:288] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:40.404206 10438 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.404855 10569 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.405164 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.405469 10307 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.405588 10637 leader_election.cc:302] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:40.405992 10701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.406167 12047 raft_consensus.cc:2781] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.406251 10701 raft_consensus.cc:465] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.406361 12047 raft_consensus.cc:686] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.406646 12047 consensus_queue.cc:237] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.407308 10700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.407552 10700 raft_consensus.cc:482] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.407691 10700 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.408932 10307 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.410728 10700 raft_consensus.cc:504] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.412572 10700 leader_election.cc:288] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.413944 10698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:40.414209 10698 raft_consensus.cc:482] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.414381 10698 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.415678 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.415997 10307 raft_consensus.cc:3032] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.416139 10307 raft_consensus.cc:729] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.416481 10307 consensus_queue.cc:260] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.416647 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.416911 10569 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.417469 10698 raft_consensus.cc:504] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.416975 10307 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.419878 10569 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.419704 10160 catalog_manager.cc:5484] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.420495 10637 leader_election.cc:302] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:40.420364 10160 catalog_manager.cc:5484] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.421079 12047 raft_consensus.cc:2781] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.421329 12047 raft_consensus.cc:686] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.421653 10307 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.421635 12047 consensus_queue.cc:237] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.422333 10699 raft_consensus.cc:1226] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Rejecting Update request from peer d7d2dd10f0d54ac08dfdb8d2a348044a for earlier term 1. Current term is 2. Ops: []
I20240503 08:13:40.424491 12005 consensus_queue.cc:1046] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.424984 12005 raft_consensus.cc:3032] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.425098 12005 raft_consensus.cc:729] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.425451 12005 consensus_queue.cc:260] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.425837 12005 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.426119 10698 leader_election.cc:288] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:40.427507 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.429240 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:40.429503 10438 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.432799 10438 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.433311 10637 leader_election.cc:302] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:40.433681 12414 raft_consensus.cc:2781] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.433944 12414 raft_consensus.cc:686] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.434262 12414 consensus_queue.cc:237] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.437487 10307 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 2.
I20240503 08:13:40.456357 10160 catalog_manager.cc:5484] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.462700 10160 catalog_manager.cc:5484] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 1 to 2, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 2 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.472826 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.473089 10438 raft_consensus.cc:482] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.473248 10438 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.475219 10438 raft_consensus.cc:504] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.476209 10438 leader_election.cc:288] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.476728 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.476723 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.476931 10698 raft_consensus.cc:3032] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.476995 10698 raft_consensus.cc:729] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.476979 10569 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.477262 10698 consensus_queue.cc:260] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.477589 10698 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.479620 10698 raft_consensus.cc:2396] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:40.479826 10569 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 3.
I20240503 08:13:40.480305 10376 leader_election.cc:302] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters: 54534138f6814b809a25c03a46e5f931
I20240503 08:13:40.480515 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.480759 10438 raft_consensus.cc:482] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.480769 11985 raft_consensus.cc:2781] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.480890 10438 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.481179 11985 raft_consensus.cc:686] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.481528 11985 consensus_queue.cc:237] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.484525 10438 raft_consensus.cc:504] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.484972 10160 catalog_manager.cc:5484] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 3 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.485919 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.485949 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.486140 10569 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.486960 10698 raft_consensus.cc:3032] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.487059 10698 raft_consensus.cc:729] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.488929 10569 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.489203 10438 leader_election.cc:288] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.489616 10376 leader_election.cc:302] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:40.489945 11985 raft_consensus.cc:2781] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.490113 11985 raft_consensus.cc:686] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.488608 10698 consensus_queue.cc:260] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.490386 11985 consensus_queue.cc:237] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.490769 10698 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.491719 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.491935 10438 raft_consensus.cc:482] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.492094 10438 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.493898 10160 catalog_manager.cc:5484] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.494630 10698 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.495083 10438 raft_consensus.cc:504] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.496102 10438 leader_election.cc:288] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.496435 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.496685 10307 raft_consensus.cc:3032] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.496768 10307 raft_consensus.cc:729] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.497048 10307 consensus_queue.cc:260] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.497432 10307 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.498036 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4670994e75a24234913724bac4c30ec4"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.498253 10438 raft_consensus.cc:482] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.498581 10438 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.498512 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.498772 10569 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.500334 10307 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.501053 10374 leader_election.cc:302] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.501384 11985 raft_consensus.cc:2781] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.501574 11985 raft_consensus.cc:686] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.501583 10438 raft_consensus.cc:504] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.501902 11985 consensus_queue.cc:237] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.502779 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.502784 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.503019 10570 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.503398 10698 raft_consensus.cc:3032] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.503492 10698 raft_consensus.cc:729] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.503858 10569 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.503808 10698 consensus_queue.cc:260] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.504175 10698 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.505926 10570 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.506592 10376 leader_election.cc:302] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:40.506879 10698 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.506994 12700 raft_consensus.cc:2781] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.507427 10438 leader_election.cc:288] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.508723 10439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.508947 10439 raft_consensus.cc:482] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.509074 10439 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.514377 10439 raft_consensus.cc:504] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.515446 10439 leader_election.cc:288] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:40.515875 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.516093 10570 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.516651 10160 catalog_manager.cc:5484] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.516994 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54e6601a404c46248232c7193aacaecb"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.517177 10440 raft_consensus.cc:482] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.517288 10440 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.518091 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.518324 10307 raft_consensus.cc:3032] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.518420 10307 raft_consensus.cc:729] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.518759 10307 consensus_queue.cc:260] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.519018 10570 raft_consensus.cc:2454] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.519193 10307 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.519541 10376 leader_election.cc:302] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:40.519946 11985 raft_consensus.cc:2781] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.520089 11985 raft_consensus.cc:686] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.520350 11985 consensus_queue.cc:237] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.520707 10440 raft_consensus.cc:504] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.521927 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.522145 10570 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.522462 10440 leader_election.cc:288] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.524845 10437 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f8e6483973f420e97503f409130cc36"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.525097 10437 raft_consensus.cc:482] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.525218 10437 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.525478 10307 raft_consensus.cc:2454] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.526593 10570 raft_consensus.cc:2454] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 3.
I20240503 08:13:40.527019 10376 leader_election.cc:302] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:40.529382 10437 raft_consensus.cc:504] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.530647 10437 leader_election.cc:288] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.531098 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.531353 10307 raft_consensus.cc:3032] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.531437 10307 raft_consensus.cc:729] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.531656 12700 raft_consensus.cc:686] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.531737 10307 consensus_queue.cc:260] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.532001 12700 consensus_queue.cc:237] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.532188 10307 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.532466 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.532727 10440 raft_consensus.cc:482] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.532907 10440 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.533452 12704 raft_consensus.cc:2781] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.533596 12704 raft_consensus.cc:686] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.533787 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.534090 10698 raft_consensus.cc:3032] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.534226 10698 raft_consensus.cc:729] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.534595 10698 consensus_queue.cc:260] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.534965 10698 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.536897 10440 raft_consensus.cc:504] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.537485 10307 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.538072 10698 raft_consensus.cc:2396] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:40.538087 10374 leader_election.cc:302] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.538419 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.538687 10308 raft_consensus.cc:3032] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.538791 10308 raft_consensus.cc:729] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.539121 10440 leader_election.cc:288] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.539153 10308 consensus_queue.cc:260] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.533922 12704 consensus_queue.cc:237] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.539747 10308 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.539122 12708 raft_consensus.cc:2781] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.540403 12708 raft_consensus.cc:686] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.540793 12708 consensus_queue.cc:237] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.542248 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.542461 10440 raft_consensus.cc:482] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.542599 10440 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.542614 10308 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.543198 10374 leader_election.cc:302] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.544661 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.544909 10569 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.544970 12705 raft_consensus.cc:2781] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.545130 12705 raft_consensus.cc:686] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.545511 12705 consensus_queue.cc:237] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.548076 10569 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.548718 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.548615 10440 raft_consensus.cc:504] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.548914 10570 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.549445 10440 leader_election.cc:288] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.551187 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.551518 10307 raft_consensus.cc:3032] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.551604 10307 raft_consensus.cc:729] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.551929 10307 consensus_queue.cc:260] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.552285 10307 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.552495 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.552769 10569 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.555490 10307 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.556075 10374 leader_election.cc:302] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.556432 12705 raft_consensus.cc:2781] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.556710 12705 raft_consensus.cc:686] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.556725 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9f91eb635604d20853c9f7518164789"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.557152 10440 raft_consensus.cc:482] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.557355 10440 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.557221 12705 consensus_queue.cc:237] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.557512 10570 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.560014 10569 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.560701 10440 raft_consensus.cc:504] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.562083 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.562332 10307 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.565572 10307 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.566226 10374 leader_election.cc:302] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.566851 10440 leader_election.cc:288] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.567653 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.567894 10569 raft_consensus.cc:3032] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.568011 10569 raft_consensus.cc:729] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:40.568418 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.568356 10569 consensus_queue.cc:260] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.568583 10440 raft_consensus.cc:482] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.568742 10440 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.568764 10569 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.569039 10160 catalog_manager.cc:5484] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.571646 10569 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.566542 12708 raft_consensus.cc:2781] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.571904 12708 raft_consensus.cc:686] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.572177 12708 consensus_queue.cc:237] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.571664 10440 raft_consensus.cc:504] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.574852 10440 leader_election.cc:288] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.575404 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.575395 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.575690 10698 raft_consensus.cc:3032] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.575721 10569 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.575803 10698 raft_consensus.cc:729] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.577412 10698 consensus_queue.cc:260] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.577879 10698 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.579105 10569 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 3.
I20240503 08:13:40.579793 10376 leader_election.cc:302] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:40.580164 12705 raft_consensus.cc:2781] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.580346 12705 raft_consensus.cc:686] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.580631 12705 consensus_queue.cc:237] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.581113 10698 raft_consensus.cc:2396] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:40.581657 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3715b584822e4cd3bee05119936125d0"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.581904 10440 raft_consensus.cc:482] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.582043 10440 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.584946 10440 raft_consensus.cc:504] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.585638 10440 leader_election.cc:288] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.586252 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.586472 10307 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.586490 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.586706 10698 raft_consensus.cc:3032] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.586791 10698 raft_consensus.cc:729] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.587092 10698 consensus_queue.cc:260] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.587489 10698 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.587769 10160 catalog_manager.cc:5484] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.588457 10160 catalog_manager.cc:5484] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.588992 10160 catalog_manager.cc:5484] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.589488 10160 catalog_manager.cc:5484] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.589727 10307 raft_consensus.cc:2454] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.590026 10160 catalog_manager.cc:5484] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 3 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.590294 10374 leader_election.cc:302] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.590538 10160 catalog_manager.cc:5484] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.591280 10698 raft_consensus.cc:2454] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.591349 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57310c72619e44c99fa7f42673648658"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.591550 10440 raft_consensus.cc:465] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.592132 12708 raft_consensus.cc:2781] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.592483 12708 raft_consensus.cc:686] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.592729 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.592936 10440 raft_consensus.cc:482] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.593099 10440 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.592944 12708 consensus_queue.cc:237] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.597465 10440 raft_consensus.cc:504] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.598954 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.599241 10698 raft_consensus.cc:3032] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.599357 10698 raft_consensus.cc:729] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.599313 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.599624 10307 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.599690 10698 consensus_queue.cc:260] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.600147 10698 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.602684 10307 raft_consensus.cc:2454] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.603149 10440 leader_election.cc:288] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:40.603317 10374 leader_election.cc:302] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.603821 12708 raft_consensus.cc:2781] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.604142 12708 raft_consensus.cc:686] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.604519 12708 consensus_queue.cc:237] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.603292 10698 raft_consensus.cc:2454] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.606022 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.606252 10440 raft_consensus.cc:482] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.606379 10440 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.606650 10160 catalog_manager.cc:5484] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 3 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.607445 10160 catalog_manager.cc:5484] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.610534 10440 raft_consensus.cc:504] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.611291 10440 leader_election.cc:288] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:40.611971 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.612211 10698 raft_consensus.cc:3032] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.612210 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.612299 10698 raft_consensus.cc:729] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.612439 10307 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.612671 10698 consensus_queue.cc:260] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.612995 10698 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.614761 10162 catalog_manager.cc:5484] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.615489 10307 raft_consensus.cc:2454] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 3.
I20240503 08:13:40.616086 10698 raft_consensus.cc:2396] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:40.616273 10374 leader_election.cc:302] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.616643 12708 raft_consensus.cc:2781] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.616829 12708 raft_consensus.cc:686] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.617143 12708 consensus_queue.cc:237] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.618301 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "205f68b8cf6d4b948576765252160224"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.618765 10440 raft_consensus.cc:482] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.620573 10440 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.624717 10440 raft_consensus.cc:504] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.626518 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.626839 10698 raft_consensus.cc:3032] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.626948 10698 raft_consensus.cc:729] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.627403 10698 consensus_queue.cc:260] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.627801 10698 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.632656 10698 raft_consensus.cc:2396] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:40.633316 10440 leader_election.cc:288] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:40.633978 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.634171 10307 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.634938 10160 catalog_manager.cc:5484] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 3 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.636194 10307 raft_consensus.cc:2454] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 3.
I20240503 08:13:40.636798 10374 leader_election.cc:302] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 54534138f6814b809a25c03a46e5f931
I20240503 08:13:40.637179 12708 raft_consensus.cc:2781] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.637420 12708 raft_consensus.cc:686] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.637861 12708 consensus_queue.cc:237] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.638044 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.638407 10440 raft_consensus.cc:482] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.638545 10440 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.641646 10440 raft_consensus.cc:504] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.642818 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.643100 10569 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.642839 10160 catalog_manager.cc:5484] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 3 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.645609 10569 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.646116 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.646276 10376 leader_election.cc:302] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:40.646368 10698 raft_consensus.cc:3032] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.646436 10698 raft_consensus.cc:729] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.646632 12708 raft_consensus.cc:2781] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.646786 10698 consensus_queue.cc:260] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.646950 12708 raft_consensus.cc:686] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.647182 10698 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.647449 12708 consensus_queue.cc:237] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.654413 10160 catalog_manager.cc:5484] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.655488 10698 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.656163 10440 leader_election.cc:288] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.657626 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.657840 10438 raft_consensus.cc:482] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.657970 10438 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.660449 10438 raft_consensus.cc:504] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.661026 10438 leader_election.cc:288] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:40.661734 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.662012 10307 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.662235 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab9203ab3c96465682075b911880b320"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.662427 10438 raft_consensus.cc:482] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.662528 10438 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.664615 10438 raft_consensus.cc:504] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.665010 10307 raft_consensus.cc:2454] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.665369 10438 leader_election.cc:288] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.665714 10374 leader_election.cc:302] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.665766 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.665966 10569 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.665963 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.666054 12708 raft_consensus.cc:2781] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.666211 10699 raft_consensus.cc:3032] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.666222 12708 raft_consensus.cc:686] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.666303 10699 raft_consensus.cc:729] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.666580 10699 consensus_queue.cc:260] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.666767 12708 consensus_queue.cc:237] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.666990 10699 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.667606 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.667870 10438 raft_consensus.cc:482] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.668056 10438 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.669260 10569 raft_consensus.cc:2454] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.669800 10376 leader_election.cc:302] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:40.670087 12705 raft_consensus.cc:2781] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.662057 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.670253 12705 raft_consensus.cc:686] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.670348 10698 raft_consensus.cc:3032] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.670432 10698 raft_consensus.cc:729] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.670537 12705 consensus_queue.cc:237] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.670617 10699 raft_consensus.cc:2454] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.670790 10698 consensus_queue.cc:260] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.671145 10698 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.674623 10698 raft_consensus.cc:2454] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.677321 10438 raft_consensus.cc:504] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.678453 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.678452 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.678683 10569 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.678683 10698 raft_consensus.cc:3032] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.678766 10698 raft_consensus.cc:729] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.679073 10698 consensus_queue.cc:260] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.679447 10698 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.681638 10569 raft_consensus.cc:2454] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.682299 10698 raft_consensus.cc:2454] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.682407 10376 leader_election.cc:302] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:40.682909 12705 raft_consensus.cc:2781] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.683137 12705 raft_consensus.cc:686] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.683456 10438 leader_election.cc:288] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.683544 12705 consensus_queue.cc:237] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.685142 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.685467 10438 raft_consensus.cc:465] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 08:13:40.685743 10569 raft_consensus.cc:1261] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.686725 12047 consensus_queue.cc:1035] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.687815 10160 catalog_manager.cc:5484] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.693361 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.693760 10438 raft_consensus.cc:482] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.693956 10438 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.697916 10438 raft_consensus.cc:504] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.700168 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.700675 10307 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.702538 10438 leader_election.cc:288] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.707190 10307 raft_consensus.cc:2454] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.707947 10374 leader_election.cc:302] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.708374 12708 raft_consensus.cc:2781] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.708519 12708 raft_consensus.cc:686] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.708798 12708 consensus_queue.cc:237] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.711504 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:40.711792 10438 raft_consensus.cc:482] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.711947 10438 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.716428 10438 raft_consensus.cc:504] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.719281 10308 raft_consensus.cc:1261] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.720602 12682 consensus_queue.cc:1035] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.722121 10438 leader_election.cc:288] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.724169 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.724439 10699 raft_consensus.cc:3032] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.724531 10699 raft_consensus.cc:729] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.724632 10307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:40.724850 10307 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.724877 10699 consensus_queue.cc:260] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.725164 10699 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.725773 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.726076 10698 raft_consensus.cc:3032] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:40.726176 10698 raft_consensus.cc:729] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.728087 10698 consensus_queue.cc:260] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.728551 10307 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 3.
I20240503 08:13:40.728642 10698 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.728794 10699 raft_consensus.cc:2396] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:40.729255 10374 leader_election.cc:302] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.729631 12700 raft_consensus.cc:2781] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.729825 12700 raft_consensus.cc:686] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.730219 12700 consensus_queue.cc:237] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.731472 12541 heartbeater.cc:503] Master 127.27.157.254:35531 was elected leader, sending a full tablet report...
I20240503 08:13:40.739972 10698 raft_consensus.cc:2454] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 2.
I20240503 08:13:40.740139 10160 catalog_manager.cc:5484] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.740773 10160 catalog_manager.cc:5484] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.749920 10164 catalog_manager.cc:5484] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 1 to 2, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 2 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.750403 10164 catalog_manager.cc:5484] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 3 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.762408 10308 raft_consensus.cc:1261] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.763166 12047 consensus_queue.cc:1035] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.770253 10569 raft_consensus.cc:1261] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.770932 12682 consensus_queue.cc:1035] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.787184 10569 raft_consensus.cc:1261] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.787881 12047 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.793406 10308 raft_consensus.cc:1261] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.795164 10569 raft_consensus.cc:1261] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.795506 12047 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.795818 12682 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.811758 10307 raft_consensus.cc:1261] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.812577 12414 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.820253 10569 raft_consensus.cc:1261] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.821234 12414 consensus_queue.cc:1035] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.836112 10307 raft_consensus.cc:1261] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.836700 12047 consensus_queue.cc:1035] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.850502 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:40.850908 10308 raft_consensus.cc:482] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.851125 10308 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.854489 10308 raft_consensus.cc:504] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.855752 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.855979 10570 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.858891 10570 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:40.859875 10245 leader_election.cc:302] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.860219 12005 raft_consensus.cc:2781] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.860363 12005 raft_consensus.cc:686] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.860769 12005 consensus_queue.cc:237] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.863293 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.863575 10698 raft_consensus.cc:3032] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.863660 10698 raft_consensus.cc:729] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.864037 10698 consensus_queue.cc:260] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.864414 10698 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.867363 10698 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:40.868463 10308 leader_election.cc:288] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.876554 10164 catalog_manager.cc:5484] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.881497 10307 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:40.881839 10307 raft_consensus.cc:482] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.881960 10307 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.884980 10307 raft_consensus.cc:504] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.885726 10307 leader_election.cc:288] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.886634 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:40.886755 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:40.886863 10438 raft_consensus.cc:3032] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.886942 10438 raft_consensus.cc:729] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.886969 10308 raft_consensus.cc:482] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.887092 10308 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.887285 10438 consensus_queue.cc:260] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.887621 10438 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.889961 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.889904 10308 raft_consensus.cc:504] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.890175 10570 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.891750 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.891995 10569 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.893162 10570 raft_consensus.cc:2454] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:40.894626 10438 raft_consensus.cc:2396] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:40.895591 10245 leader_election.cc:302] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:40.895835 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.896123 10698 raft_consensus.cc:3032] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.896237 10698 raft_consensus.cc:729] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.896652 10698 consensus_queue.cc:260] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.897162 10698 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.895238 10569 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:40.899518 12546 raft_consensus.cc:2781] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.899737 12546 raft_consensus.cc:686] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.899977 10245 leader_election.cc:302] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.900418 10308 leader_election.cc:288] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.900225 12546 consensus_queue.cc:237] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.900849 12754 raft_consensus.cc:2781] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.900981 12754 raft_consensus.cc:686] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.901324 12754 consensus_queue.cc:237] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.903230 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4764c2955533410188d40d9465156bcc"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:40.903467 10308 raft_consensus.cc:482] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.903620 10308 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.904992 10698 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:40.905944 10308 raft_consensus.cc:504] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.906451 10308 leader_election.cc:288] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.906976 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.907191 10698 raft_consensus.cc:3032] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.907269 10698 raft_consensus.cc:729] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.907390 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.907613 10569 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.907582 10698 consensus_queue.cc:260] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.907946 10698 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.909919 10569 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:40.910604 10698 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:40.910626 10245 leader_election.cc:302] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.910907 12546 raft_consensus.cc:2781] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.911103 12546 raft_consensus.cc:686] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.911458 12546 consensus_queue.cc:237] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:40.911592 10164 catalog_manager.cc:5484] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:40.912169 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "205f68b8cf6d4b948576765252160224"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:40.912469 10308 raft_consensus.cc:482] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.912197 10164 catalog_manager.cc:5484] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.912601 10308 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:40.916752 10308 raft_consensus.cc:504] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.917757 10308 leader_election.cc:288] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 4 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:40.918339 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:40.918622 10438 raft_consensus.cc:3032] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:40.918727 10438 raft_consensus.cc:729] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.917999 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.919001 10698 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:40.919111 10569 raft_consensus.cc:1261] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.919030 10438 consensus_queue.cc:260] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.919387 10438 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:40.919816 12705 consensus_queue.cc:1035] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.922367 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:40.923578 10308 raft_consensus.cc:482] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.924432 10698 raft_consensus.cc:2396] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:40.924173 10164 catalog_manager.cc:5484] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.924682 10308 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.925384 10438 raft_consensus.cc:2396] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:40.926046 10242 leader_election.cc:302] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:40.926527 12754 raft_consensus.cc:2726] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:13:40.928982 10308 raft_consensus.cc:504] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.930053 10570 raft_consensus.cc:1261] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.930271 10308 leader_election.cc:288] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:40.930680 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.930886 10570 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.931165 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:40.931407 10438 raft_consensus.cc:3032] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.931504 10438 raft_consensus.cc:729] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.931546 10307 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9f91eb635604d20853c9f7518164789"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:40.931748 10307 raft_consensus.cc:482] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.931883 10307 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.931774 10438 consensus_queue.cc:260] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.932065 10438 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.933987 10570 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:40.934839 10438 raft_consensus.cc:2396] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:40.935213 12708 consensus_queue.cc:1035] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.935740 10307 raft_consensus.cc:504] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.936393 10245 leader_election.cc:302] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:40.936753 12754 raft_consensus.cc:2781] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.936918 12754 raft_consensus.cc:686] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.937062 10307 leader_election.cc:288] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:40.937240 12754 consensus_queue.cc:237] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.939445 10568 raft_consensus.cc:1261] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:13:40.940224 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:40.941092 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d01df9a11104a33a7971e133310acce"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:40.941376 10309 raft_consensus.cc:482] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.941553 10309 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.940531 10438 raft_consensus.cc:3032] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.947640 10438 raft_consensus.cc:729] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:40.941416 12700 consensus_queue.cc:1035] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20240503 08:13:40.945859 10309 raft_consensus.cc:504] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.950174 10569 raft_consensus.cc:1261] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.951067 12047 consensus_queue.cc:1035] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.948365 10438 consensus_queue.cc:260] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.952427 10438 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.957738 10438 raft_consensus.cc:2396] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:40.968876 10309 leader_election.cc:288] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:40.969820 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.970091 10571 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.970670 10698 raft_consensus.cc:1261] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.972520 10308 raft_consensus.cc:1261] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.973309 12704 consensus_queue.cc:1035] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.973732 10571 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:40.974684 12704 consensus_queue.cc:1035] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.977350 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57310c72619e44c99fa7f42673648658"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:40.977649 10309 raft_consensus.cc:482] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:40.977787 10309 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.980476 10245 leader_election.cc:302] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.980737 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:40.980927 12546 raft_consensus.cc:2781] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.981091 12546 raft_consensus.cc:686] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.981078 10698 raft_consensus.cc:3032] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:40.981485 12546 consensus_queue.cc:237] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.981637 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.981858 10570 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.981997 10698 raft_consensus.cc:729] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:40.982702 10698 consensus_queue.cc:260] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:40.983273 10698 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:40.986198 10701 raft_consensus.cc:1261] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:13:40.986366 10701 pending_rounds.cc:76] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 3
I20240503 08:13:40.986392 10570 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:40.986449 10701 pending_rounds.cc:97] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:13:40.986613 10701 raft_consensus.cc:2866] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:40.987159 10245 leader_election.cc:302] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:40.987596 12005 raft_consensus.cc:2781] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:40.987759 12005 raft_consensus.cc:686] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.987709 11985 consensus_queue.cc:1035] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:40.988088 12005 consensus_queue.cc:237] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:40.991111 10309 raft_consensus.cc:504] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:40.991240 10569 raft_consensus.cc:1261] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:13:40.991835 10309 leader_election.cc:288] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 2 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:40.992012 10569 raft_consensus.cc:1261] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:40.993351 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:40.993638 10569 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:40.993626 10698 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:40.997205 10569 raft_consensus.cc:2454] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 2.
I20240503 08:13:40.997804 10245 leader_election.cc:302] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:40.998306 12754 raft_consensus.cc:2781] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:40.998093 10160 catalog_manager.cc:5484] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:40.998523 12754 raft_consensus.cc:686] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:40.998821 12754 consensus_queue.cc:237] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.002759 11985 consensus_queue.cc:1035] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.003544 11985 consensus_queue.cc:1035] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.006492 10569 raft_consensus.cc:1261] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.022504 10308 raft_consensus.cc:1261] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.023260 12414 consensus_queue.cc:1035] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.025238 10698 raft_consensus.cc:1261] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.025980 12708 consensus_queue.cc:1035] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.031191 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.031525 10438 raft_consensus.cc:3032] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:41.031669 10438 raft_consensus.cc:729] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.032063 10438 consensus_queue.cc:260] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.032341 12704 consensus_queue.cc:1035] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.032672 10438 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:41.032742 10571 raft_consensus.cc:1261] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.034229 12704 consensus_queue.cc:1035] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.034361 10701 raft_consensus.cc:1261] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:13:41.034507 10701 pending_rounds.cc:76] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 3
I20240503 08:13:41.034579 10701 pending_rounds.cc:97] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:13:41.034669 10701 raft_consensus.cc:2866] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:41.036557 11985 consensus_queue.cc:1035] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.037459 10438 raft_consensus.cc:2454] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 2.
I20240503 08:13:41.047652 10564 raft_consensus.cc:1261] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:13:41.051276 10164 catalog_manager.cc:5484] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 1 to 2, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 2 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.052160 10164 catalog_manager.cc:5484] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:41.052506 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.052765 10309 raft_consensus.cc:482] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.052901 10309 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.052891 10164 catalog_manager.cc:5484] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.056434 12778 consensus_queue.cc:1035] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.058476 10307 raft_consensus.cc:1261] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.059635 10699 raft_consensus.cc:1261] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:13:41.059888 10699 pending_rounds.cc:76] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 3
I20240503 08:13:41.060001 10699 pending_rounds.cc:97] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:13:41.060163 10699 raft_consensus.cc:2866] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:41.060173 12704 consensus_queue.cc:1035] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.062975 10309 raft_consensus.cc:504] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.063812 10309 leader_election.cc:288] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:41.064968 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8127b76db73402d8100faad68271644"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.065088 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.065182 10308 raft_consensus.cc:482] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.065292 10308 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:41.065364 10438 raft_consensus.cc:3032] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.065479 10438 raft_consensus.cc:729] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.065798 10438 consensus_queue.cc:260] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.066157 10306 raft_consensus.cc:1261] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.066293 10438 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.061728 10701 raft_consensus.cc:1261] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.067792 10306 raft_consensus.cc:1261] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.068217 12780 consensus_queue.cc:1035] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.068387 11985 consensus_queue.cc:1035] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.069034 11985 consensus_queue.cc:1035] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.072662 10308 raft_consensus.cc:504] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.074138 10438 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.075093 10308 leader_election.cc:288] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 2 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.075423 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.075753 10700 raft_consensus.cc:3032] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:41.075873 10700 raft_consensus.cc:729] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:41.076269 10700 consensus_queue.cc:260] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.076783 10700 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:41.076721 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.077262 10309 raft_consensus.cc:482] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.077476 10309 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.081188 10700 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 2.
I20240503 08:13:41.081974 10242 leader_election.cc:302] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.082438 12754 raft_consensus.cc:2781] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:41.082720 12754 raft_consensus.cc:686] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.083160 10245 leader_election.cc:302] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.083114 12754 consensus_queue.cc:237] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.083578 12005 raft_consensus.cc:2781] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.083748 12005 raft_consensus.cc:686] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.084038 12005 consensus_queue.cc:237] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.085971 10309 raft_consensus.cc:504] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.087713 12779 consensus_queue.cc:1035] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.089043 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.089362 10438 raft_consensus.cc:3032] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.089661 10438 raft_consensus.cc:729] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.090072 10438 consensus_queue.cc:260] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.090525 10438 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.093809 10438 raft_consensus.cc:2396] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:41.094745 10309 leader_election.cc:288] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.096349 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.096504 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.096678 10308 raft_consensus.cc:482] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.096824 10308 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.097432 10700 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.099957 10308 raft_consensus.cc:504] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.101408 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.101773 10697 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.103427 10308 leader_election.cc:288] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.104772 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.105094 10309 raft_consensus.cc:482] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.105290 10309 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.106776 10697 raft_consensus.cc:2454] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.107582 10242 leader_election.cc:302] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.107954 12546 raft_consensus.cc:2781] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.108131 12546 raft_consensus.cc:686] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.108409 12546 consensus_queue.cc:237] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.109853 10700 raft_consensus.cc:2454] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.110576 10242 leader_election.cc:302] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:41.110991 12005 raft_consensus.cc:2781] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.113644 10700 raft_consensus.cc:1261] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:13:41.113808 10700 pending_rounds.cc:76] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 3
I20240503 08:13:41.113906 10700 pending_rounds.cc:97] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:13:41.114009 10700 raft_consensus.cc:2866] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:41.114894 12779 consensus_queue.cc:1035] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.125447 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:41.125788 10570 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:41.128074 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:41.131717 10570 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 2.
I20240503 08:13:41.136514 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.136792 10438 raft_consensus.cc:3032] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.136888 10438 raft_consensus.cc:729] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.137187 10438 consensus_queue.cc:260] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.137861 12005 raft_consensus.cc:686] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.138262 12005 consensus_queue.cc:237] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.139356 10438 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.141789 10567 raft_consensus.cc:1261] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.142993 10438 raft_consensus.cc:2396] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:41.144757 10700 raft_consensus.cc:1261] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.146296 10309 raft_consensus.cc:504] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.147397 12780 consensus_queue.cc:1035] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.147835 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.148048 12780 consensus_queue.cc:1035] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.148063 10700 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.154518 10162 catalog_manager.cc:5484] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 1 to 2, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 2 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:41.155372 10162 catalog_manager.cc:5484] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.147315 10309 leader_election.cc:288] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 4 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.147468 10308 raft_consensus.cc:1226] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Rejecting Update request from peer 2b3a6ff465524577a400de6af7e70f71 for earlier term 3. Current term is 4. Ops: []
I20240503 08:13:41.160269 12708 consensus_queue.cc:1046] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.160807 12708 raft_consensus.cc:3032] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.160962 12708 raft_consensus.cc:729] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.161379 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.161376 12708 consensus_queue.cc:260] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.161957 12708 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.162434 10571 raft_consensus.cc:1261] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.162755 10700 raft_consensus.cc:2396] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:41.165481 12779 consensus_queue.cc:1035] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.166430 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3715b584822e4cd3bee05119936125d0"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.166698 10305 raft_consensus.cc:482] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.166867 10305 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.167448 10438 raft_consensus.cc:2396] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:41.168817 10245 leader_election.cc:302] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:41.169400 12005 raft_consensus.cc:2726] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:13:41.168515 10162 catalog_manager.cc:5484] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.174332 10305 raft_consensus.cc:504] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.174582 10162 catalog_manager.cc:5484] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.176033 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.176343 10700 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.176888 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.177196 10438 raft_consensus.cc:3032] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.177335 10438 raft_consensus.cc:729] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.177681 10438 consensus_queue.cc:260] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.178157 10438 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.179029 10698 raft_consensus.cc:1261] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.179731 12708 consensus_queue.cc:1035] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.179682 10700 raft_consensus.cc:2454] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.181486 10438 raft_consensus.cc:2454] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.181603 10242 leader_election.cc:302] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.182078 12005 raft_consensus.cc:2781] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.182255 12005 raft_consensus.cc:686] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.182605 12005 consensus_queue.cc:237] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.184789 10305 leader_election.cc:288] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.186897 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.187134 10305 raft_consensus.cc:482] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.187258 10305 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.190057 10305 raft_consensus.cc:504] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.192102 10309 raft_consensus.cc:1261] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:13:41.192836 12708 consensus_queue.cc:1035] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.197616 10305 leader_election.cc:288] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.198001 10567 raft_consensus.cc:1261] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.198926 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.199198 10700 raft_consensus.cc:3032] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.199309 10700 raft_consensus.cc:729] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:41.199671 10700 consensus_queue.cc:260] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.200111 10700 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.203821 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:41.204192 10571 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.207887 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.208287 10305 raft_consensus.cc:482] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.208503 10305 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:41.208670 10571 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.211411 10700 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.212002 10242 leader_election.cc:302] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.212401 10305 raft_consensus.cc:504] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.212494 12005 raft_consensus.cc:2781] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.212805 12005 raft_consensus.cc:686] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.213083 12780 consensus_queue.cc:1035] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.213102 12005 consensus_queue.cc:237] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.214928 10700 raft_consensus.cc:1261] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:13:41.215045 10700 pending_rounds.cc:76] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 3
I20240503 08:13:41.215101 10700 pending_rounds.cc:97] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:13:41.215168 10700 raft_consensus.cc:2866] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:41.216022 12779 consensus_queue.cc:1035] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.222101 10161 catalog_manager.cc:5484] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.228937 10305 leader_election.cc:288] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 2 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.229765 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:41.230005 10567 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:41.231127 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f8e6483973f420e97503f409130cc36"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.231411 10305 raft_consensus.cc:482] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.231556 10305 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.233654 10567 raft_consensus.cc:2454] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 2.
I20240503 08:13:41.234241 12673 heartbeater.cc:503] Master 127.27.157.254:35531 was elected leader, sending a full tablet report...
I20240503 08:13:41.234311 10245 leader_election.cc:302] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.234663 12754 raft_consensus.cc:2781] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:13:41.234704 10305 raft_consensus.cc:504] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.235029 12754 raft_consensus.cc:686] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.235285 12754 consensus_queue.cc:237] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.235563 10305 leader_election.cc:288] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.236747 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.236948 10305 raft_consensus.cc:482] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.237090 10305 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.237540 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:41.237200 10168 catalog_manager.cc:5484] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:41.238054 10567 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.241254 10567 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.241276 10700 raft_consensus.cc:1261] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.242107 12705 consensus_queue.cc:1035] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.243125 10305 raft_consensus.cc:504] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.244796 10245 leader_election.cc:302] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.245179 12005 raft_consensus.cc:2781] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.245338 12005 raft_consensus.cc:686] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.245602 12005 consensus_queue.cc:237] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.246516 10305 leader_election.cc:288] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:41.247743 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.247812 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:41.247975 10305 raft_consensus.cc:482] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.248080 10305 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.248234 10571 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.250000 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.250335 10438 raft_consensus.cc:3032] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:13:41.250447 10438 raft_consensus.cc:729] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.250804 10438 consensus_queue.cc:260] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.251315 10438 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:13:41.251376 10305 raft_consensus.cc:504] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.252261 10305 leader_election.cc:288] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:41.252645 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.252897 10698 raft_consensus.cc:3032] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.252981 10698 raft_consensus.cc:729] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:41.253248 10698 consensus_queue.cc:260] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.253508 10698 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.255267 10438 raft_consensus.cc:2454] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 2.
I20240503 08:13:41.255791 10698 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.256209 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.256489 10438 raft_consensus.cc:3032] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.256537 10242 leader_election.cc:302] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.256506 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.256583 10438 raft_consensus.cc:729] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.256727 10305 raft_consensus.cc:482] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.256879 12546 raft_consensus.cc:2781] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.257015 12546 raft_consensus.cc:686] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.256980 10438 consensus_queue.cc:260] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.257442 10438 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.257299 12546 consensus_queue.cc:237] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.258129 10571 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.258761 10305 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.260143 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.260389 10440 raft_consensus.cc:3032] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.260501 10440 raft_consensus.cc:729] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.260546 10245 leader_election.cc:302] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.260824 12005 raft_consensus.cc:2781] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.258575 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:41.260982 12005 raft_consensus.cc:686] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.260849 10440 consensus_queue.cc:260] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.261127 10567 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.261339 10440 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.261265 12005 consensus_queue.cc:237] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.263731 10305 raft_consensus.cc:504] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.264768 10567 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.265069 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.265403 10698 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.265389 10440 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.268675 10698 raft_consensus.cc:2454] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.269742 10242 leader_election.cc:302] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.270128 12754 raft_consensus.cc:2781] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.270371 12754 raft_consensus.cc:686] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.270689 12754 consensus_queue.cc:237] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.270843 10160 catalog_manager.cc:5484] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 1 to 2, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 2 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.272256 10305 leader_election.cc:288] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 3 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.273630 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:41.273880 10309 raft_consensus.cc:482] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.274045 10309 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.276229 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.276492 10440 raft_consensus.cc:3032] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.276611 10440 raft_consensus.cc:729] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.277009 10440 consensus_queue.cc:260] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.277400 10440 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.280751 10309 raft_consensus.cc:504] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.281961 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.282171 10698 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.282780 10440 raft_consensus.cc:2396] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:41.285459 10309 leader_election.cc:288] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 4 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.290598 10698 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 4.
I20240503 08:13:41.292001 10242 leader_election.cc:302] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.292368 12546 raft_consensus.cc:2781] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.292553 12546 raft_consensus.cc:686] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.292834 10438 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 3.
I20240503 08:13:41.292912 12546 consensus_queue.cc:237] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.300710 10168 catalog_manager.cc:5484] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.301992 10168 catalog_manager.cc:5484] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.302749 10168 catalog_manager.cc:5484] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 3 to 4, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 4 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.303339 10168 catalog_manager.cc:5484] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.303913 10168 catalog_manager.cc:5484] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 3 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.310973 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.311671 10438 raft_consensus.cc:3032] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.312058 10438 raft_consensus.cc:729] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.312546 10438 consensus_queue.cc:260] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.313529 10438 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.329538 10438 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 4.
I20240503 08:13:41.335834 10700 raft_consensus.cc:1261] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.337013 12754 consensus_queue.cc:1035] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.345479 10567 raft_consensus.cc:1261] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.346750 12797 consensus_queue.cc:1035] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.360451 10567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.360813 10567 raft_consensus.cc:482] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.360992 10567 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.366389 10571 raft_consensus.cc:1261] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.367880 12797 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.371953 10700 raft_consensus.cc:1261] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.372649 12754 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.374460 10567 raft_consensus.cc:504] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.376945 10567 leader_election.cc:288] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.376983 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.377230 10438 raft_consensus.cc:3032] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.377184 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.377326 10438 raft_consensus.cc:729] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.377388 10700 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.377636 10438 consensus_queue.cc:260] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.378046 10438 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.379031 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.379228 10571 raft_consensus.cc:482] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.379472 10571 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.380414 10700 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.380908 10504 leader_election.cc:302] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:41.380961 10438 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.381306 12783 raft_consensus.cc:2781] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.381460 12783 raft_consensus.cc:686] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.381775 12783 consensus_queue.cc:237] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.383498 10571 raft_consensus.cc:504] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.385586 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.385844 10438 raft_consensus.cc:3032] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.385967 10438 raft_consensus.cc:729] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.386286 10438 consensus_queue.cc:260] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.386515 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.386688 10438 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.386752 10700 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.386739 10571 leader_election.cc:288] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.388317 10440 raft_consensus.cc:1261] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:13:41.388435 10440 pending_rounds.cc:76] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71: Aborting all ops after (but not including) 3
I20240503 08:13:41.388521 10440 pending_rounds.cc:97] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:13:41.388644 10440 raft_consensus.cc:2866] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:41.389161 10570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.389367 10570 raft_consensus.cc:482] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.389505 10570 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.389549 12754 consensus_queue.cc:1035] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.389860 10438 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.390322 10507 leader_election.cc:302] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:41.390669 12785 raft_consensus.cc:2781] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.390831 12785 raft_consensus.cc:686] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.391224 12785 consensus_queue.cc:237] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.392820 10570 raft_consensus.cc:504] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.394096 10569 raft_consensus.cc:1261] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:13:41.394707 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.394997 10698 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.394920 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.395169 10309 raft_consensus.cc:3032] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.395244 10309 raft_consensus.cc:729] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.395529 10309 consensus_queue.cc:260] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.395895 10309 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.396977 12546 consensus_queue.cc:1035] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.398350 10569 raft_consensus.cc:1261] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.398753 10309 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.399055 10570 leader_election.cc:288] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:41.399632 10505 leader_election.cc:302] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.399715 12546 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.400326 10701 raft_consensus.cc:1261] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.400745 12783 raft_consensus.cc:2781] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.400936 12783 raft_consensus.cc:686] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.401252 12783 consensus_queue.cc:237] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.403867 10566 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4670994e75a24234913724bac4c30ec4"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.404059 10566 raft_consensus.cc:482] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.404198 10566 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.404393 10701 raft_consensus.cc:1261] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.404871 12754 consensus_queue.cc:1035] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.405699 12797 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.407845 10566 raft_consensus.cc:504] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.408766 10566 leader_election.cc:288] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.409068 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.409279 10440 raft_consensus.cc:3032] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.409384 10440 raft_consensus.cc:729] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.409646 10440 consensus_queue.cc:260] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.410038 10440 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.411760 10566 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.411849 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.412086 10697 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.412106 10566 raft_consensus.cc:482] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.412300 10566 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.412793 10440 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.413232 10438 raft_consensus.cc:1261] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:13:41.413348 10438 pending_rounds.cc:76] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71: Aborting all ops after (but not including) 3
I20240503 08:13:41.413419 10438 pending_rounds.cc:97] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:13:41.413517 10438 raft_consensus.cc:2866] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:41.413652 10700 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.414361 12754 consensus_queue.cc:1035] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.414374 10507 leader_election.cc:302] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:41.414824 12783 raft_consensus.cc:2781] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.414985 12783 raft_consensus.cc:686] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.415355 12783 consensus_queue.cc:237] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.417083 10698 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.421108 10697 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.430230 10566 raft_consensus.cc:504] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.430824 10570 raft_consensus.cc:1226] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Rejecting Update request from peer d7d2dd10f0d54ac08dfdb8d2a348044a for earlier term 3. Current term is 4. Ops: []
I20240503 08:13:41.432277 12754 consensus_queue.cc:1046] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.433054 12005 raft_consensus.cc:3032] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.433187 12005 raft_consensus.cc:729] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.433586 12005 consensus_queue.cc:260] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.434139 12005 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.387146 10567 raft_consensus.cc:1261] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:13:41.440047 12005 consensus_queue.cc:1035] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.443794 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.444259 10309 raft_consensus.cc:2396] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:13:41.444780 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.445082 10697 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.451050 10697 raft_consensus.cc:2396] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:13:41.451799 10504 leader_election.cc:302] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2; no voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:41.452221 12783 raft_consensus.cc:2726] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:13:41.453706 10566 leader_election.cc:288] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.455670 10570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4764c2955533410188d40d9465156bcc"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.456017 10570 raft_consensus.cc:482] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.456180 10570 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.461454 10570 raft_consensus.cc:504] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.462967 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.463342 10309 raft_consensus.cc:3032] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.463487 10309 raft_consensus.cc:729] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.463878 10309 consensus_queue.cc:260] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.464370 10309 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.468652 10309 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.469489 10505 leader_election.cc:302] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.470374 12783 raft_consensus.cc:2781] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.470706 12783 raft_consensus.cc:686] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.471168 12783 consensus_queue.cc:237] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.475006 10440 raft_consensus.cc:1261] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:13:41.475136 10440 pending_rounds.cc:76] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71: Aborting all ops after (but not including) 3
I20240503 08:13:41.475201 10440 pending_rounds.cc:97] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:13:41.475257 10440 raft_consensus.cc:2866] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:41.476047 12005 consensus_queue.cc:1035] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.476290 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.476626 10697 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.480948 10567 raft_consensus.cc:1261] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.481768 12005 consensus_queue.cc:1035] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.484097 10566 raft_consensus.cc:1261] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:13:41.484673 12005 consensus_queue.cc:1035] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.491218 10570 leader_election.cc:288] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.493604 10164 catalog_manager.cc:5484] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.493824 10568 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.494097 10568 raft_consensus.cc:482] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.494252 10568 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.494467 10164 catalog_manager.cc:5484] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 3 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.495601 10164 catalog_manager.cc:5484] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:41.496235 10698 raft_consensus.cc:1261] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:13:41.496152 10164 catalog_manager.cc:5484] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 3 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.497057 12005 consensus_queue.cc:1035] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.500937 10568 raft_consensus.cc:504] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.502444 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.502745 10309 raft_consensus.cc:3032] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.502858 10309 raft_consensus.cc:729] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.503309 10309 consensus_queue.cc:260] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.503829 10309 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.504277 10697 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.507014 10309 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.507947 10505 leader_election.cc:302] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.508880 12783 raft_consensus.cc:2781] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.509044 12783 raft_consensus.cc:686] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.509308 12783 consensus_queue.cc:237] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.512208 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.512439 10440 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.514496 10438 raft_consensus.cc:1261] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:13:41.514613 10438 pending_rounds.cc:76] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71: Aborting all ops after (but not including) 3
I20240503 08:13:41.514667 10438 pending_rounds.cc:97] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:13:41.514742 10438 raft_consensus.cc:2866] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:41.515296 10440 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.516346 10438 raft_consensus.cc:1261] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.516325 10568 leader_election.cc:288] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.516378 10697 raft_consensus.cc:1261] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.517374 12005 consensus_queue.cc:1035] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 08:13:41.517482 10567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9f91eb635604d20853c9f7518164789"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.517742 10567 raft_consensus.cc:482] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.517863 10567 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.519412 10164 catalog_manager.cc:5484] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.520056 10164 catalog_manager.cc:5484] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.521339 10567 raft_consensus.cc:504] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.521780 10549 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 08:13:41.523556 12797 consensus_queue.cc:1035] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.523607 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.523821 10438 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.523900 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.524130 10309 raft_consensus.cc:3032] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.524215 10309 raft_consensus.cc:729] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.524302 12797 consensus_queue.cc:1035] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.524495 10309 consensus_queue.cc:260] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.524853 10309 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.526852 10438 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.527552 10507 leader_election.cc:302] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:41.527791 10309 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.527930 12785 raft_consensus.cc:2781] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.528079 12785 raft_consensus.cc:686] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.528373 12785 consensus_queue.cc:237] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.537416 12345 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:41.537561 10162 catalog_manager.cc:5484] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.544675 10440 raft_consensus.cc:1261] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:13:41.544844 10440 pending_rounds.cc:76] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71: Aborting all ops after (but not including) 3
I20240503 08:13:41.544905 10440 pending_rounds.cc:97] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:13:41.544989 10440 raft_consensus.cc:2866] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:41.545892 12005 consensus_queue.cc:1035] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.549178 10697 raft_consensus.cc:1261] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:13:41.549814 12546 consensus_queue.cc:1035] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.554873 10567 leader_election.cc:288] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.556820 10570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d01df9a11104a33a7971e133310acce"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.557096 10570 raft_consensus.cc:482] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.557256 10570 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.559938 10566 raft_consensus.cc:1261] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.560739 12005 consensus_queue.cc:1035] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.563233 10570 raft_consensus.cc:504] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.564739 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.565063 10698 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.569216 10698 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.570009 10504 leader_election.cc:302] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:41.570451 12783 raft_consensus.cc:2781] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.570614 12783 raft_consensus.cc:686] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.570659 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.570955 10309 raft_consensus.cc:3032] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.571054 10309 raft_consensus.cc:729] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.570981 12783 consensus_queue.cc:237] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.571471 10309 consensus_queue.cc:260] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.571987 10309 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.575716 10570 leader_election.cc:288] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:41.577260 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.577526 10571 raft_consensus.cc:482] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.577672 10571 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.577993 10309 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.580948 10571 raft_consensus.cc:504] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.585451 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.585738 10440 raft_consensus.cc:3032] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.585831 10440 raft_consensus.cc:729] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.586154 10440 consensus_queue.cc:260] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.586562 10440 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.591938 10440 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.593561 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.593801 10700 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.596835 10700 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.597358 10504 leader_election.cc:302] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:41.597700 12785 raft_consensus.cc:2781] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.597885 10571 leader_election.cc:288] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.598342 10162 catalog_manager.cc:5484] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.597890 12785 raft_consensus.cc:686] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.599848 12785 consensus_queue.cc:237] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.605685 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57310c72619e44c99fa7f42673648658"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.605923 10571 raft_consensus.cc:482] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.606066 10571 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.608301 10162 catalog_manager.cc:5484] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.617964 10571 raft_consensus.cc:504] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.619558 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.619870 10309 raft_consensus.cc:3032] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.619971 10309 raft_consensus.cc:729] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.620061 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.620283 10440 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.620316 10309 consensus_queue.cc:260] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.620772 10309 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.623195 10440 raft_consensus.cc:2454] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.625579 10507 leader_election.cc:302] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:41.626019 12785 raft_consensus.cc:2781] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.626205 12785 raft_consensus.cc:686] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.626510 12785 consensus_queue.cc:237] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.630038 10162 catalog_manager.cc:5484] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 2 to 3, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 3 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:41.631350 10571 leader_election.cc:288] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.633167 10309 raft_consensus.cc:2454] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.645393 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.645713 10571 raft_consensus.cc:482] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.645845 10571 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.649236 10571 raft_consensus.cc:504] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.651361 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.651763 10440 raft_consensus.cc:2379] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 in current term 3: Already voted for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in this term.
I20240503 08:13:41.653206 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.653652 10309 raft_consensus.cc:2379] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 in current term 3: Already voted for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in this term.
I20240503 08:13:41.654232 10505 leader_election.cc:302] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2; no voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:41.654671 12783 raft_consensus.cc:2726] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240503 08:13:41.657629 10567 raft_consensus.cc:1261] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.658788 12546 consensus_queue.cc:1035] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.663878 10571 leader_election.cc:288] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:41.667236 10700 raft_consensus.cc:1261] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.668419 12546 consensus_queue.cc:1035] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.670329 10567 raft_consensus.cc:1261] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.677222 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8127b76db73402d8100faad68271644"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.677525 10571 raft_consensus.cc:482] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.677709 10571 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.677670 12797 consensus_queue.cc:1035] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.679646 10418 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 08:13:41.685914 10571 raft_consensus.cc:504] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.687463 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.687717 10700 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.691208 10438 raft_consensus.cc:1261] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 08:13:41.691255 10439 raft_consensus.cc:1261] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.691390 10438 pending_rounds.cc:76] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71: Aborting all ops after (but not including) 3
I20240503 08:13:41.691473 10438 pending_rounds.cc:97] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 08:13:41.691578 10438 raft_consensus.cc:2866] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:41.693368 10440 raft_consensus.cc:1261] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.694079 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.694295 12754 consensus_queue.cc:1035] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.694466 10309 raft_consensus.cc:3032] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.694576 10309 raft_consensus.cc:729] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.694743 12797 consensus_queue.cc:1035] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.694785 12754 consensus_queue.cc:1035] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.694958 10309 consensus_queue.cc:260] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.697376 10700 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.697963 10504 leader_election.cc:302] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:41.698292 12783 raft_consensus.cc:2781] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.698428 12783 raft_consensus.cc:686] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.698693 12783 consensus_queue.cc:237] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.699553 10309 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.704275 10700 raft_consensus.cc:1261] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 08:13:41.706189 12546 consensus_queue.cc:1035] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.708281 10439 raft_consensus.cc:1261] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:41.710160 10160 catalog_manager.cc:5484] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 2 to 3, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 3 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.712246 10698 raft_consensus.cc:1261] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.716517 12546 consensus_queue.cc:1035] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.717332 10309 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.718655 12797 consensus_queue.cc:1035] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.721081 10438 raft_consensus.cc:1261] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.721648 12005 consensus_queue.cc:1035] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.724252 10698 raft_consensus.cc:1261] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:41.725091 12754 consensus_queue.cc:1035] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.731200 10566 raft_consensus.cc:1261] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 08:13:41.733983 10571 leader_election.cc:288] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:41.734372 10566 raft_consensus.cc:1261] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.735890 10566 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab9203ab3c96465682075b911880b320"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.736099 10440 raft_consensus.cc:1261] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.736142 10566 raft_consensus.cc:482] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.736297 10566 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.736990 12005 consensus_queue.cc:1035] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.737277 12546 consensus_queue.cc:1035] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 08:13:41.737461 12005 consensus_queue.cc:1035] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.745527 10566 raft_consensus.cc:504] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.747562 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.747880 10698 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.752183 10698 raft_consensus.cc:2454] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.752933 10504 leader_election.cc:302] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:41.753453 12785 raft_consensus.cc:2781] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.753464 10697 raft_consensus.cc:1261] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.753640 12785 raft_consensus.cc:686] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.754002 12785 consensus_queue.cc:237] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.754137 12797 consensus_queue.cc:1035] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.758030 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.758313 10440 raft_consensus.cc:3032] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.758780 10440 raft_consensus.cc:729] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.759147 10440 consensus_queue.cc:260] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.759856 10440 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.763741 10571 raft_consensus.cc:1261] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.764288 10566 leader_election.cc:288] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.764400 12797 consensus_queue.cc:1035] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.765507 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.765758 10571 raft_consensus.cc:482] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.765935 10571 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.775938 10162 catalog_manager.cc:5484] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 3 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.779424 10438 raft_consensus.cc:1261] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.780316 12797 consensus_queue.cc:1035] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.784036 10571 raft_consensus.cc:504] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.785557 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.785894 10438 raft_consensus.cc:3032] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.786005 10438 raft_consensus.cc:729] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.786373 10438 consensus_queue.cc:260] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.786652 10566 raft_consensus.cc:1261] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.786883 10438 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.787513 12005 consensus_queue.cc:1035] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.790453 10700 raft_consensus.cc:1261] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.791196 12783 consensus_queue.cc:1035] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.793550 10438 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.794142 10507 leader_election.cc:302] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:41.794577 12783 raft_consensus.cc:2781] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.794783 12783 raft_consensus.cc:686] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.795174 12783 consensus_queue.cc:237] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.810599 10438 raft_consensus.cc:1261] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.811367 12783 consensus_queue.cc:1035] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.820607 10162 catalog_manager.cc:5484] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 3 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.825277 10287 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 08:13:41.826650 10440 raft_consensus.cc:2454] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.832150 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.832423 10697 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.835806 12608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:41.835906 10697 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.836477 10571 leader_election.cc:288] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.837808 10566 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.838042 10566 raft_consensus.cc:482] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.838183 10566 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.838991 10679 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:41.840961 12213 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:41.841085 10566 raft_consensus.cc:504] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.842278 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.842526 10700 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.845140 10700 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.845613 10504 leader_election.cc:302] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:41.845994 12783 raft_consensus.cc:2781] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.846129 10566 leader_election.cc:288] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:41.846176 12783 raft_consensus.cc:686] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.846499 12783 consensus_queue.cc:237] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.847920 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.848165 10309 raft_consensus.cc:3032] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.848131 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.848248 10309 raft_consensus.cc:729] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.848325 10571 raft_consensus.cc:482] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.848445 10571 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.848529 10309 consensus_queue.cc:260] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:41.848940 10309 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.851455 10571 raft_consensus.cc:504] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.852269 10571 leader_election.cc:288] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.853631 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.853873 10440 raft_consensus.cc:3032] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.853984 10440 raft_consensus.cc:729] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:41.854261 10440 consensus_queue.cc:260] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.854455 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.854657 10440 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.854671 10697 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.855108 10162 catalog_manager.cc:5484] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.857461 10697 raft_consensus.cc:2454] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.857558 10440 raft_consensus.cc:2454] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.858122 10507 leader_election.cc:302] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:41.858498 10700 raft_consensus.cc:1261] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240503 08:13:41.858498 12784 raft_consensus.cc:2781] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.858642 12784 raft_consensus.cc:686] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.859052 10309 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.859026 12784 consensus_queue.cc:237] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.859447 12785 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:41.863531 10160 catalog_manager.cc:5484] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 2 to 3, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 3 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.866024 10697 raft_consensus.cc:1261] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.866741 10309 raft_consensus.cc:1261] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240503 08:13:41.866894 12785 consensus_queue.cc:1035] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.867565 12785 consensus_queue.cc:1035] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:41.873220 10440 raft_consensus.cc:1261] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:41.873246 10305 raft_consensus.cc:1261] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240503 08:13:41.875119 12784 consensus_queue.cc:1035] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.876019 12784 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:41.879206 10698 raft_consensus.cc:1261] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240503 08:13:41.880226 12783 consensus_queue.cc:1035] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:41.881208 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.881472 10571 raft_consensus.cc:482] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.881599 10571 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.884681 10571 raft_consensus.cc:504] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.900167 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.900527 10309 raft_consensus.cc:3032] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:13:41.900682 10309 raft_consensus.cc:729] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.901057 10309 consensus_queue.cc:260] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.901552 10309 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.904604 10309 raft_consensus.cc:2454] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.905208 10505 leader_election.cc:302] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.905638 12784 raft_consensus.cc:2781] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:13:41.905807 12784 raft_consensus.cc:686] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.906116 12784 consensus_queue.cc:237] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.907538 10571 leader_election.cc:288] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 3 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.909947 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.910202 10440 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:13:41.910387 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f8e6483973f420e97503f409130cc36"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.910693 10571 raft_consensus.cc:482] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.910862 10571 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.914234 10440 raft_consensus.cc:2454] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 3.
I20240503 08:13:41.915210 12476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:41.915583 10438 raft_consensus.cc:1261] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:41.916201 12784 consensus_queue.cc:1035] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.919190 10571 raft_consensus.cc:504] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.920656 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.920748 10697 raft_consensus.cc:1261] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:41.920883 10309 raft_consensus.cc:3032] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.920997 10309 raft_consensus.cc:729] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.921324 12785 consensus_queue.cc:1035] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:41.921298 10309 consensus_queue.cc:260] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.921717 10309 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.924888 10309 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.925365 10505 leader_election.cc:302] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.925716 12785 raft_consensus.cc:2781] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.925882 12785 raft_consensus.cc:686] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.925727 10164 catalog_manager.cc:5484] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 2 to 3, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 3 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.926143 12785 consensus_queue.cc:237] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:41.928141 10571 leader_election.cc:288] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.929411 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.929610 10571 raft_consensus.cc:482] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.929747 10571 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.932315 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.932605 10440 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.935696 10440 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.937525 10162 catalog_manager.cc:5484] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:41.938896 10571 raft_consensus.cc:504] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.943807 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.944064 10309 raft_consensus.cc:3032] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.944164 10309 raft_consensus.cc:729] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.944418 10571 leader_election.cc:288] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:41.944489 10309 consensus_queue.cc:260] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.944912 10309 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.945657 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.945935 10571 raft_consensus.cc:482] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.946110 10571 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.953447 10571 raft_consensus.cc:504] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.954205 10309 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.943661 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:41.954598 10438 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.954778 10505 leader_election.cc:302] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:41.955109 12784 raft_consensus.cc:2781] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.955262 12784 raft_consensus.cc:686] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.955957 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.956218 10697 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.957558 10438 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.958940 10697 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.960120 12784 consensus_queue.cc:237] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.961726 10504 leader_election.cc:302] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:41.962144 12784 raft_consensus.cc:2781] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:41.962275 12784 raft_consensus.cc:686] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:41.962524 12784 consensus_queue.cc:237] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.965072 10309 raft_consensus.cc:1261] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240503 08:13:41.965404 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:41.965596 10305 raft_consensus.cc:3032] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:41.965678 10305 raft_consensus.cc:729] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:41.965674 12785 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:41.965914 10305 consensus_queue.cc:260] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:41.966310 10305 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.970445 10697 raft_consensus.cc:1261] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240503 08:13:41.971695 10162 catalog_manager.cc:5484] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:41.973162 12784 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:41.978989 10305 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:41.979463 10571 leader_election.cc:288] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:41.987174 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54e6601a404c46248232c7193aacaecb"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.987450 10571 raft_consensus.cc:482] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.987586 10571 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.993045 10571 raft_consensus.cc:504] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:41.993166 10160 catalog_manager.cc:5484] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:41.996243 10571 leader_election.cc:288] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:41.997431 10566 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:41.997665 10566 raft_consensus.cc:482] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:41.997658 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:41.997771 10566 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:41.997912 10697 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.000794 10566 raft_consensus.cc:504] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.001101 10697 raft_consensus.cc:2454] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:42.002162 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.002399 10305 raft_consensus.cc:3032] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.002494 10305 raft_consensus.cc:729] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.002692 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.002784 10305 consensus_queue.cc:260] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.002941 10439 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.003175 10305 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.005949 10439 raft_consensus.cc:2454] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:42.006139 10305 raft_consensus.cc:2454] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:42.006551 10438 raft_consensus.cc:1261] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:42.006722 10505 leader_election.cc:302] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.007113 12784 raft_consensus.cc:2781] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.007262 12784 raft_consensus.cc:686] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.007598 12784 consensus_queue.cc:237] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.008173 12785 consensus_queue.cc:1035] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.009056 10566 leader_election.cc:288] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.009256 10504 leader_election.cc:302] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.009562 12783 raft_consensus.cc:2781] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.009691 12783 raft_consensus.cc:686] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.009969 12783 consensus_queue.cc:237] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.014346 10698 raft_consensus.cc:1261] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:42.015187 12785 consensus_queue.cc:1035] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.022666 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.023073 10440 raft_consensus.cc:3032] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.023263 10440 raft_consensus.cc:729] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.023866 10697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.024129 10697 raft_consensus.cc:482] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.024376 10697 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.026623 10571 raft_consensus.cc:1226] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Rejecting Update request from peer 2b3a6ff465524577a400de6af7e70f71 for earlier term 3. Current term is 4. Ops: []
I20240503 08:13:42.028172 10697 raft_consensus.cc:504] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.028811 10160 catalog_manager.cc:5484] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.029472 10160 catalog_manager.cc:5484] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 3 to 4, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 4 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.029974 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.030227 10438 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.034735 10438 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.035290 10637 leader_election.cc:302] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.035665 12047 raft_consensus.cc:2781] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.035797 12047 raft_consensus.cc:686] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.036093 12047 consensus_queue.cc:237] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.039062 10697 leader_election.cc:288] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.039152 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.039465 10571 raft_consensus.cc:3032] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.023643 10440 consensus_queue.cc:260] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.039939 12780 consensus_queue.cc:1046] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.040177 12780 consensus_queue.cc:1225] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Peer 22588cae35464393b2d6adb27e3114b2 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240503 08:13:42.039584 10571 raft_consensus.cc:729] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.040778 10571 consensus_queue.cc:260] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.041208 10571 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.040881 10697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.041260 10440 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.041602 10697 raft_consensus.cc:482] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.043112 10697 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.043310 10305 raft_consensus.cc:1261] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:42.044297 12784 consensus_queue.cc:1035] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.046511 10440 raft_consensus.cc:2454] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 4.
I20240503 08:13:42.050963 10697 raft_consensus.cc:504] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.054602 10440 raft_consensus.cc:1261] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:42.059412 12783 consensus_queue.cc:1035] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.059649 10438 raft_consensus.cc:1261] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:42.061666 12783 consensus_queue.cc:1035] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.064733 10160 catalog_manager.cc:5484] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.069737 10571 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.073122 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.073375 10440 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.074354 10305 raft_consensus.cc:1261] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:42.074637 10697 leader_election.cc:288] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.075464 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.075762 10571 raft_consensus.cc:3032] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.075891 10571 raft_consensus.cc:729] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.076093 12784 consensus_queue.cc:1035] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.076156 10571 consensus_queue.cc:260] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.076562 10571 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.077118 10698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.077332 10698 raft_consensus.cc:482] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.077435 10698 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.079519 10571 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.080379 10637 leader_election.cc:302] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.080703 12414 raft_consensus.cc:2781] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.080901 12414 raft_consensus.cc:686] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.081167 12414 consensus_queue.cc:237] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.082247 10698 raft_consensus.cc:504] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.083011 10698 leader_election.cc:288] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.083859 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.084062 10305 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.084867 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.085069 10571 raft_consensus.cc:3032] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.085175 10571 raft_consensus.cc:729] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.085448 10571 consensus_queue.cc:260] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.086010 10571 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.087033 10305 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.087399 10700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4670994e75a24234913724bac4c30ec4"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.087625 10700 raft_consensus.cc:482] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.087716 10700 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.089798 10571 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.090528 10637 leader_election.cc:302] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.090605 10700 raft_consensus.cc:504] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.094036 10440 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.094707 12414 raft_consensus.cc:2781] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.094900 12414 raft_consensus.cc:686] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.095199 12414 consensus_queue.cc:237] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.097476 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.097759 10440 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.098435 10700 leader_election.cc:288] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.099581 10700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.099835 10700 raft_consensus.cc:482] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.100008 10700 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.100850 10161 catalog_manager.cc:5484] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 4, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 4 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.102699 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.102998 10571 raft_consensus.cc:3032] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.103145 10571 raft_consensus.cc:729] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.103575 10571 consensus_queue.cc:260] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.104111 10571 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.105875 10700 raft_consensus.cc:504] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.108007 10571 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.108258 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.108654 10305 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.108681 10637 leader_election.cc:302] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.108999 12414 raft_consensus.cc:2781] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.109156 12414 raft_consensus.cc:686] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.109439 12414 consensus_queue.cc:237] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.111992 10305 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.112560 10635 leader_election.cc:302] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.112874 12047 raft_consensus.cc:2781] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.113021 12047 raft_consensus.cc:686] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.113274 12047 consensus_queue.cc:237] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.113269 10161 catalog_manager.cc:5484] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.114745 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.115059 10571 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.118069 10571 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.118706 10700 leader_election.cc:288] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.121291 10700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4764c2955533410188d40d9465156bcc"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.121553 10700 raft_consensus.cc:482] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.121675 10700 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.126533 10440 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.128778 10161 catalog_manager.cc:5484] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 4, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 4 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.129920 10161 catalog_manager.cc:5484] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 5, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.131547 10700 raft_consensus.cc:504] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.132369 10700 leader_election.cc:288] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.133203 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.133440 10305 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.133523 10700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54e6601a404c46248232c7193aacaecb"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.133736 10700 raft_consensus.cc:482] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.133946 10700 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.136533 10305 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.137076 10635 leader_election.cc:302] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.137552 12414 raft_consensus.cc:2781] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.137722 12414 raft_consensus.cc:686] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.138113 12414 consensus_queue.cc:237] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.139221 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.139518 10571 raft_consensus.cc:3032] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.139603 10571 raft_consensus.cc:729] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.139894 10571 consensus_queue.cc:260] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.140277 10571 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.143533 10571 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.143525 10161 catalog_manager.cc:5484] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.145609 10700 raft_consensus.cc:504] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.146498 10700 leader_election.cc:288] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.147245 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.147511 10440 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.147636 10697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.147805 10697 raft_consensus.cc:482] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.147893 10697 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.149852 10697 raft_consensus.cc:504] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.150517 10697 leader_election.cc:288] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.150971 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.150996 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.151216 10571 raft_consensus.cc:3032] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.151213 10566 raft_consensus.cc:3032] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.151341 10571 raft_consensus.cc:729] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.151393 10566 raft_consensus.cc:729] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.151671 10566 consensus_queue.cc:260] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.152019 10566 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.152083 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.152298 10438 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.152581 10697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8127b76db73402d8100faad68271644"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.152832 10697 raft_consensus.cc:482] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.152983 10697 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.155097 10566 raft_consensus.cc:2396] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:13:42.155133 10438 raft_consensus.cc:2454] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.155786 10637 leader_election.cc:302] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.151674 10571 consensus_queue.cc:260] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.156142 10571 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.156193 12047 raft_consensus.cc:2781] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.156345 12047 raft_consensus.cc:686] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.156576 12047 consensus_queue.cc:237] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.150449 10440 raft_consensus.cc:2454] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.158322 10697 raft_consensus.cc:504] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.159085 10637 leader_election.cc:302] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.159415 12414 raft_consensus.cc:2781] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.159557 12414 raft_consensus.cc:686] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.159790 12414 consensus_queue.cc:237] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.160135 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.160393 10566 raft_consensus.cc:3032] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.160552 10566 raft_consensus.cc:729] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.160920 10566 consensus_queue.cc:260] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.160815 10161 catalog_manager.cc:5484] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 4, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 4 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.161293 10566 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.162012 10571 raft_consensus.cc:2396] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.164148 10566 raft_consensus.cc:2396] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:13:42.165011 10697 leader_election.cc:288] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.165369 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.165596 10305 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.166107 10698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.166297 10698 raft_consensus.cc:482] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.166445 10698 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.169288 10698 raft_consensus.cc:504] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.168612 10305 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.170028 10698 leader_election.cc:288] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.170423 10635 leader_election.cc:302] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 22588cae35464393b2d6adb27e3114b2
I20240503 08:13:42.170677 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.170900 10309 raft_consensus.cc:3032] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.171000 10309 raft_consensus.cc:729] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.171281 10309 consensus_queue.cc:260] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.171591 12414 raft_consensus.cc:2781] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.171684 10309 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.171628 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.171801 12414 raft_consensus.cc:686] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.172097 10440 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.172166 12414 consensus_queue.cc:237] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.175191 10309 raft_consensus.cc:2454] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.175217 10440 raft_consensus.cc:2454] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.175683 10635 leader_election.cc:302] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.175997 12047 raft_consensus.cc:2781] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.176151 12047 raft_consensus.cc:686] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.176409 12047 consensus_queue.cc:237] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.176242 10161 catalog_manager.cc:5484] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.176692 10700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab9203ab3c96465682075b911880b320"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.176882 10700 raft_consensus.cc:482] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.176988 10700 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.176895 10161 catalog_manager.cc:5484] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 4, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 4 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.180039 10700 raft_consensus.cc:504] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.180711 10700 leader_election.cc:288] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.181598 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.181820 10440 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.182699 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.182780 10697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.182965 10697 raft_consensus.cc:482] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.183001 10566 raft_consensus.cc:3032] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.183123 10697 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.183252 10566 raft_consensus.cc:729] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.183604 10566 consensus_queue.cc:260] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.183974 10566 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.183779 10161 catalog_manager.cc:5484] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 4 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.186403 10697 raft_consensus.cc:504] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.187043 10566 raft_consensus.cc:2396] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:13:42.187090 10440 raft_consensus.cc:2454] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.187505 10697 leader_election.cc:288] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.187714 10637 leader_election.cc:302] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.187893 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.188166 10309 raft_consensus.cc:3032] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.188256 10309 raft_consensus.cc:729] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.188558 10309 consensus_queue.cc:260] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.188920 10700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.188980 10309 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.189124 10700 raft_consensus.cc:482] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.189267 10700 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.189491 12414 raft_consensus.cc:2781] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.189762 12414 raft_consensus.cc:686] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.190064 12414 consensus_queue.cc:237] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.192471 10700 raft_consensus.cc:504] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.193220 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.193538 10438 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.193696 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.193917 10439 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.197464 10439 raft_consensus.cc:2454] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.198204 10637 leader_election.cc:302] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.198791 12047 raft_consensus.cc:2781] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.198835 10438 raft_consensus.cc:2454] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.199609 10637 leader_election.cc:302] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.199832 10309 raft_consensus.cc:2454] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.200774 10697 raft_consensus.cc:1261] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:42.201593 12852 consensus_queue.cc:1035] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.202627 10440 raft_consensus.cc:1261] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:42.203411 12852 consensus_queue.cc:1035] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.204229 12866 raft_consensus.cc:2781] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.204439 12866 raft_consensus.cc:686] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.204927 12866 consensus_queue.cc:237] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.207715 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.208082 10309 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.212858 10309 raft_consensus.cc:2454] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.213681 10700 leader_election.cc:288] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.216437 10698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3715b584822e4cd3bee05119936125d0"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.216702 10698 raft_consensus.cc:482] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.216812 10698 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.221257 10698 raft_consensus.cc:504] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.221647 12047 raft_consensus.cc:686] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.223562 10439 raft_consensus.cc:1261] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:42.224812 12852 consensus_queue.cc:1035] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.229001 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.229283 10309 raft_consensus.cc:3032] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.229429 10309 raft_consensus.cc:729] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.229773 10309 consensus_queue.cc:260] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.230265 10309 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.235715 10309 raft_consensus.cc:2454] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.236418 10635 leader_election.cc:302] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.222724 12047 consensus_queue.cc:237] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.237900 10309 raft_consensus.cc:1261] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:42.238016 10698 leader_election.cc:288] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.238592 12414 raft_consensus.cc:2781] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.238794 12414 raft_consensus.cc:686] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.239086 12414 consensus_queue.cc:237] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.240967 10162 catalog_manager.cc:5484] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 4, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 4 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.242175 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.242410 10439 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.243222 10698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.243468 10698 raft_consensus.cc:482] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.243604 10698 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.244853 12783 consensus_queue.cc:1035] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.250456 10439 raft_consensus.cc:2454] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.251152 10698 raft_consensus.cc:504] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.252616 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.252873 10566 raft_consensus.cc:3032] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.253006 10566 raft_consensus.cc:729] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.253398 10566 consensus_queue.cc:260] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.253816 10566 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.255726 10698 leader_election.cc:288] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.256558 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.256796 10305 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.257025 10162 catalog_manager.cc:5484] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 5, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.257354 10566 raft_consensus.cc:2396] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:42.257681 10162 catalog_manager.cc:5484] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 4 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.257958 10698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "205f68b8cf6d4b948576765252160224"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.258164 10698 raft_consensus.cc:482] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.258260 10698 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.258257 10162 catalog_manager.cc:5484] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 4 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.259667 10305 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.261024 10698 raft_consensus.cc:504] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.261164 10635 leader_election.cc:302] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 22588cae35464393b2d6adb27e3114b2
I20240503 08:13:42.261647 12047 raft_consensus.cc:2781] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.261947 12047 raft_consensus.cc:686] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.262188 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.262220 12047 consensus_queue.cc:237] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.262439 10439 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.264329 10698 leader_election.cc:288] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.265084 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.265461 10305 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.265425 10698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.265609 10698 raft_consensus.cc:482] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.265741 10698 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.266598 10439 raft_consensus.cc:2396] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:13:42.266791 10161 catalog_manager.cc:5484] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.268535 10698 raft_consensus.cc:504] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.269265 10698 leader_election.cc:288] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.269402 10305 raft_consensus.cc:2454] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.269793 10635 leader_election.cc:302] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:42.270242 12047 raft_consensus.cc:2781] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.270296 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.270391 12047 raft_consensus.cc:686] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.270524 10439 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.270689 12047 consensus_queue.cc:237] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.271239 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.271507 10566 raft_consensus.cc:3032] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.271605 10566 raft_consensus.cc:729] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.271855 10566 consensus_queue.cc:260] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.272264 10566 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.274469 10439 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.275035 10566 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.275513 10637 leader_election.cc:302] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.275842 12047 raft_consensus.cc:2781] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.276010 12047 raft_consensus.cc:686] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.276238 12047 consensus_queue.cc:237] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.276751 10698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.276970 10698 raft_consensus.cc:482] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.277076 10698 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.279096 10698 raft_consensus.cc:504] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.279695 10698 leader_election.cc:288] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.280407 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.280622 10439 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.281082 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.281302 10566 raft_consensus.cc:3032] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.281411 10566 raft_consensus.cc:729] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.281679 10566 consensus_queue.cc:260] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.282065 10566 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.282891 10439 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.283425 10637 leader_election.cc:302] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.283880 12047 raft_consensus.cc:2781] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.284152 12047 raft_consensus.cc:686] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.284483 12047 consensus_queue.cc:237] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.284806 10566 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.284776 10161 catalog_manager.cc:5484] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 5, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.285399 10161 catalog_manager.cc:5484] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 4, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 4 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.291126 10161 catalog_manager.cc:5484] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.295511 10698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d01df9a11104a33a7971e133310acce"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.295807 10698 raft_consensus.cc:482] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.295945 10698 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.298902 10698 raft_consensus.cc:504] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.300267 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.300307 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.300506 10305 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.300578 10566 raft_consensus.cc:3032] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.300690 10566 raft_consensus.cc:729] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.300971 10566 consensus_queue.cc:260] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.301406 10566 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.303745 10305 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.304309 10635 leader_election.cc:302] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.304665 12047 raft_consensus.cc:2781] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.304682 10566 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.304826 12047 raft_consensus.cc:686] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.305102 12047 consensus_queue.cc:237] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.307045 10698 leader_election.cc:288] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.308475 10161 catalog_manager.cc:5484] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.309326 10698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.309549 10698 raft_consensus.cc:482] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.309662 10698 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.315519 10698 raft_consensus.cc:504] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.316843 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.317126 10305 raft_consensus.cc:3032] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.317245 10305 raft_consensus.cc:729] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.317250 10309 raft_consensus.cc:1261] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:42.317605 10305 consensus_queue.cc:260] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.318022 10305 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.317955 12783 consensus_queue.cc:1035] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.320238 10698 leader_election.cc:288] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.320616 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.320839 10439 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.324539 10440 raft_consensus.cc:1261] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 08:13:42.324652 10698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.324873 10698 raft_consensus.cc:482] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.325006 10698 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.325627 12784 consensus_queue.cc:1035] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.330427 10439 raft_consensus.cc:2454] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.330929 10637 leader_election.cc:302] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:42.331347 12865 raft_consensus.cc:2781] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.331516 12865 raft_consensus.cc:686] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.331820 12865 consensus_queue.cc:237] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.333402 10698 raft_consensus.cc:504] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.333624 10305 raft_consensus.cc:2454] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 4.
I20240503 08:13:42.334064 10698 leader_election.cc:288] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.335676 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.336026 10305 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.336455 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.336675 10566 raft_consensus.cc:3032] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.336813 10566 raft_consensus.cc:729] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.337127 10566 consensus_queue.cc:260] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.337466 10566 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.338094 10161 catalog_manager.cc:5484] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 4 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.339742 10697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:42.339963 10697 raft_consensus.cc:482] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.340114 10697 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.340644 10566 raft_consensus.cc:2396] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:42.342809 10697 raft_consensus.cc:504] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.344198 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.344419 10309 raft_consensus.cc:3032] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.344537 10309 raft_consensus.cc:729] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.344897 10305 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.344869 10309 consensus_queue.cc:260] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.345292 10309 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.345870 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.346088 10439 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.348470 10309 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.349205 10635 leader_election.cc:302] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.349565 12047 raft_consensus.cc:2781] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.349617 10635 leader_election.cc:302] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 22588cae35464393b2d6adb27e3114b2
I20240503 08:13:42.349745 12047 raft_consensus.cc:686] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.349931 12865 raft_consensus.cc:2781] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.350109 12865 raft_consensus.cc:686] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.350039 12047 consensus_queue.cc:237] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.350215 10439 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 5.
I20240503 08:13:42.350426 12865 consensus_queue.cc:237] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.350808 10697 leader_election.cc:288] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 5 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.353487 10161 catalog_manager.cc:5484] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 4 to 5, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.355504 10439 raft_consensus.cc:1261] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240503 08:13:42.357324 12784 consensus_queue.cc:1035] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:42.361629 10305 raft_consensus.cc:1261] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Refusing update from remote peer 22588cae35464393b2d6adb27e3114b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240503 08:13:42.361685 10161 catalog_manager.cc:5484] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 5 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.362236 12783 consensus_queue.cc:1035] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:42.366384 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.366598 10440 raft_consensus.cc:482] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.366708 10440 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.371074 10440 raft_consensus.cc:504] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.372423 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.372422 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.372653 10566 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.372673 10697 raft_consensus.cc:3032] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.372792 10697 raft_consensus.cc:729] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.373160 10697 consensus_queue.cc:260] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.373464 10697 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.375461 10697 raft_consensus.cc:2396] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:13:42.375741 10566 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 6.
I20240503 08:13:42.376277 10376 leader_election.cc:302] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters: 54534138f6814b809a25c03a46e5f931
I20240503 08:13:42.376708 12708 raft_consensus.cc:2781] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:42.376864 12708 raft_consensus.cc:686] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.377173 12708 consensus_queue.cc:237] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.376708 10440 leader_election.cc:288] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.379860 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.380060 10440 raft_consensus.cc:482] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.380180 10440 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.380786 10161 catalog_manager.cc:5484] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 5 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 6 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.383095 10440 raft_consensus.cc:504] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.384434 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.384701 10566 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.384876 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.385083 10697 raft_consensus.cc:3032] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.385151 10697 raft_consensus.cc:729] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.385437 10697 consensus_queue.cc:260] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.385725 10697 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.388345 10566 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.388935 10376 leader_election.cc:302] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:42.389246 12780 raft_consensus.cc:2781] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.389433 12780 raft_consensus.cc:686] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.389741 12780 consensus_queue.cc:237] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.389034 10697 raft_consensus.cc:2396] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:42.391048 10440 leader_election.cc:288] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.392131 10439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.392340 10439 raft_consensus.cc:482] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.392442 10439 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.393649 10161 catalog_manager.cc:5484] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.395728 10439 raft_consensus.cc:504] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.397486 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.397706 10439 leader_election.cc:288] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 4 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.397421 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.397890 10309 raft_consensus.cc:3032] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.397918 10566 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.398036 10309 raft_consensus.cc:729] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.398401 10309 consensus_queue.cc:260] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.398842 10309 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.399127 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4670994e75a24234913724bac4c30ec4"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.399363 10438 raft_consensus.cc:482] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.399478 10438 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.402276 10438 raft_consensus.cc:504] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.402485 10309 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 4.
I20240503 08:13:42.403314 10374 leader_election.cc:302] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.403793 12708 raft_consensus.cc:2781] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.403906 10438 leader_election.cc:288] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.403972 12708 raft_consensus.cc:686] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 4 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.404265 12708 consensus_queue.cc:237] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.405334 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.405565 10697 raft_consensus.cc:3032] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.405642 10697 raft_consensus.cc:729] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.405932 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.406095 10438 raft_consensus.cc:482] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.406204 10438 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.408504 10161 catalog_manager.cc:5484] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 3 to 4, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 4 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.409452 10697 consensus_queue.cc:260] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.409788 10697 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.404585 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.412079 10571 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.414577 10571 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.414937 10566 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 4.
I20240503 08:13:42.415098 10376 leader_election.cc:302] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:42.415499 12780 raft_consensus.cc:2781] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.415673 12780 raft_consensus.cc:686] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.415964 12780 consensus_queue.cc:237] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.416772 10697 raft_consensus.cc:2396] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:42.417131 10438 raft_consensus.cc:504] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.418226 10438 leader_election.cc:288] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.419037 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.419238 10566 raft_consensus.cc:3032] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.419309 10566 raft_consensus.cc:729] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.419602 10566 consensus_queue.cc:260] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.419870 10566 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.421633 10161 catalog_manager.cc:5484] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.423148 10566 raft_consensus.cc:2396] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.423533 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54e6601a404c46248232c7193aacaecb"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.423820 10438 raft_consensus.cc:482] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.423954 10438 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.426748 10438 raft_consensus.cc:504] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.427373 10438 leader_election.cc:288] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.427805 10440 raft_consensus.cc:1226] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Rejecting Update request from peer 22588cae35464393b2d6adb27e3114b2 for earlier term 4. Current term is 5. Ops: []
I20240503 08:13:42.428308 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.428478 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f8e6483973f420e97503f409130cc36"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.428548 10309 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.428647 10440 raft_consensus.cc:482] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.428596 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.428758 10440 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.428798 10697 raft_consensus.cc:3032] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.428865 10697 raft_consensus.cc:729] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.429301 10697 consensus_queue.cc:260] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.429630 10697 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.431550 10309 raft_consensus.cc:2454] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.431535 10440 raft_consensus.cc:504] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.432909 10697 raft_consensus.cc:2396] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.433648 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.433681 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.433884 10571 raft_consensus.cc:3032] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.434038 10571 raft_consensus.cc:729] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.434084 10566 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.434389 10571 consensus_queue.cc:260] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.434780 10571 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.437599 10566 raft_consensus.cc:2396] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.438532 10374 leader_election.cc:302] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 22588cae35464393b2d6adb27e3114b2
I20240503 08:13:42.439056 12780 raft_consensus.cc:2781] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.439229 12780 raft_consensus.cc:686] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.439566 12780 consensus_queue.cc:237] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.440558 10571 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.441000 10440 leader_election.cc:288] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.441385 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.442147 10376 leader_election.cc:302] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71; no voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:42.442524 12780 raft_consensus.cc:2726] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:13:42.442646 10376 leader_election.cc:302] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:42.443130 10309 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.443686 12780 raft_consensus.cc:2781] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.443825 12780 raft_consensus.cc:686] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.444084 12780 consensus_queue.cc:237] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.445477 10161 catalog_manager.cc:5484] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.451648 10309 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.452147 10164 catalog_manager.cc:5484] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.458802 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.459228 10440 raft_consensus.cc:482] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.459437 10440 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.462358 10440 raft_consensus.cc:504] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.463570 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.463798 10309 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.464215 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.464427 10571 raft_consensus.cc:3032] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.464530 10571 raft_consensus.cc:729] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.464876 10571 consensus_queue.cc:260] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.465241 10571 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.467368 10309 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.467927 10374 leader_election.cc:302] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.468516 12708 raft_consensus.cc:2781] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.468672 12708 raft_consensus.cc:686] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.468742 10440 leader_election.cc:288] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.468977 12708 consensus_queue.cc:237] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.470933 10571 raft_consensus.cc:2396] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:42.471413 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.471751 10440 raft_consensus.cc:482] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.471984 10440 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.473217 10161 catalog_manager.cc:5484] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.474784 10440 raft_consensus.cc:504] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.476130 10440 leader_election.cc:288] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.477198 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9f91eb635604d20853c9f7518164789"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.477401 10438 raft_consensus.cc:482] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.477526 10438 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.477808 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.478008 10571 raft_consensus.cc:3032] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.478104 10571 raft_consensus.cc:729] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.478389 10571 consensus_queue.cc:260] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.478778 10571 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.480307 10438 raft_consensus.cc:504] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.481314 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.481568 10309 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.481487 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.481791 10305 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.482266 10571 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.484442 10309 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.485570 10305 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.485639 10376 leader_election.cc:302] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:42.486086 12780 raft_consensus.cc:2781] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.486274 12780 raft_consensus.cc:686] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.486533 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.486559 12780 consensus_queue.cc:237] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.486759 10571 raft_consensus.cc:3032] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.486858 10571 raft_consensus.cc:729] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.487105 10571 consensus_queue.cc:260] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.487555 10571 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.487968 10374 leader_election.cc:302] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.488242 12708 raft_consensus.cc:2781] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.488382 12708 raft_consensus.cc:686] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.488636 12708 consensus_queue.cc:237] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.490869 10438 leader_election.cc:288] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.491715 10571 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.492071 10439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.492237 10439 raft_consensus.cc:482] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.492339 10439 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.493923 10161 catalog_manager.cc:5484] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.494589 10161 catalog_manager.cc:5484] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.495100 10439 raft_consensus.cc:504] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.496815 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.497061 10697 raft_consensus.cc:3032] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.497136 10697 raft_consensus.cc:729] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.497421 10697 consensus_queue.cc:260] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.497676 10697 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.500662 10697 raft_consensus.cc:2396] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:13:42.501446 10439 leader_election.cc:288] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.502534 10439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3715b584822e4cd3bee05119936125d0"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.502741 10439 raft_consensus.cc:482] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.502838 10439 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.505717 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.505779 10439 raft_consensus.cc:504] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.506009 10571 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.507088 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.507295 10305 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.508065 10309 raft_consensus.cc:1261] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:13:42.508798 12414 consensus_queue.cc:1035] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:42.510097 10305 raft_consensus.cc:2454] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.511209 10374 leader_election.cc:302] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.511555 12780 raft_consensus.cc:2781] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.511679 12780 raft_consensus.cc:686] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.511984 12780 consensus_queue.cc:237] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.514081 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.514321 10697 raft_consensus.cc:3032] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.514411 10697 raft_consensus.cc:729] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.514659 10697 consensus_queue.cc:260] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.514909 10697 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.518007 10697 raft_consensus.cc:2396] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:42.517846 10160 catalog_manager.cc:5484] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.518939 10439 leader_election.cc:288] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.520622 10439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57310c72619e44c99fa7f42673648658"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.520918 10439 raft_consensus.cc:482] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.521250 10439 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.524369 10571 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 6.
I20240503 08:13:42.525187 10376 leader_election.cc:302] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters: 54534138f6814b809a25c03a46e5f931
I20240503 08:13:42.525854 12708 raft_consensus.cc:2781] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:42.526042 12708 raft_consensus.cc:686] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.526468 12708 consensus_queue.cc:237] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.526851 10439 raft_consensus.cc:504] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.527565 10439 leader_election.cc:288] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 4 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.528707 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.528936 10440 raft_consensus.cc:482] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.529059 10440 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.530985 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.531220 10566 raft_consensus.cc:3032] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.531754 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.531839 10440 raft_consensus.cc:504] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.532007 10305 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.532531 10440 leader_election.cc:288] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.532792 10161 catalog_manager.cc:5484] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 5 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 6 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.533231 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.533459 10308 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.534924 10305 raft_consensus.cc:2454] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 4.
I20240503 08:13:42.535218 10566 raft_consensus.cc:729] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.535575 10566 consensus_queue.cc:260] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.535993 10566 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.536285 10308 raft_consensus.cc:2454] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.537144 10374 leader_election.cc:302] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.537531 10374 leader_election.cc:302] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.538743 12780 raft_consensus.cc:2781] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.538725 10566 raft_consensus.cc:2454] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 4.
I20240503 08:13:42.539405 12890 raft_consensus.cc:2781] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.539907 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.540239 10440 raft_consensus.cc:482] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.540460 10440 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.540405 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.540715 10697 raft_consensus.cc:3032] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.540831 10697 raft_consensus.cc:729] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.541165 10697 consensus_queue.cc:260] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.541493 10697 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.542579 12780 raft_consensus.cc:686] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 4 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.543015 12780 consensus_queue.cc:237] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.546829 12890 raft_consensus.cc:686] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.547179 10697 raft_consensus.cc:2396] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.547215 12890 consensus_queue.cc:237] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.549055 10566 raft_consensus.cc:1261] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:13:42.545192 10440 raft_consensus.cc:504] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.550882 12866 consensus_queue.cc:1035] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.551360 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.551581 10697 raft_consensus.cc:3032] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.551991 10697 raft_consensus.cc:729] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.552346 10697 consensus_queue.cc:260] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.552636 10697 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.555202 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.555634 10308 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.555720 10697 raft_consensus.cc:2396] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.556504 10440 leader_election.cc:288] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.557735 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "205f68b8cf6d4b948576765252160224"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.557981 10440 raft_consensus.cc:482] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.558115 10440 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.560951 10440 raft_consensus.cc:504] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.561756 10440 leader_election.cc:288] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.562657 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.562903 10305 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.563843 10308 raft_consensus.cc:2454] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 6.
I20240503 08:13:42.564505 10374 leader_election.cc:302] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 54534138f6814b809a25c03a46e5f931
I20240503 08:13:42.564882 12891 raft_consensus.cc:2781] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:42.564985 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.565073 12891 raft_consensus.cc:686] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.565223 10697 raft_consensus.cc:3032] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.565305 10697 raft_consensus.cc:729] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.565421 12891 consensus_queue.cc:237] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.565594 10697 consensus_queue.cc:260] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.565778 10305 raft_consensus.cc:2454] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 6.
I20240503 08:13:42.565879 10697 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.566426 10374 leader_election.cc:302] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.567008 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.567196 12708 raft_consensus.cc:2781] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:42.567415 12708 raft_consensus.cc:686] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.567682 12708 consensus_queue.cc:237] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.567200 10440 raft_consensus.cc:482] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.568099 10440 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.564013 10309 raft_consensus.cc:1261] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:42.569308 10697 raft_consensus.cc:2396] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.570190 12047 consensus_queue.cc:1035] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.574896 10439 raft_consensus.cc:1261] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 08:13:42.575855 12047 consensus_queue.cc:1035] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.578820 10440 raft_consensus.cc:504] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.580049 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.580332 10566 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.583411 10566 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.583956 10376 leader_election.cc:302] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:42.584309 12708 raft_consensus.cc:2781] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.584585 12708 raft_consensus.cc:686] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.585062 12708 consensus_queue.cc:237] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.586086 10440 leader_election.cc:288] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.587188 10438 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.587463 10438 raft_consensus.cc:482] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.587587 10438 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.590469 10438 raft_consensus.cc:504] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.591557 10438 leader_election.cc:288] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.591657 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.591908 10698 raft_consensus.cc:3032] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.591991 10698 raft_consensus.cc:729] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.591993 10440 raft_consensus.cc:1226] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Rejecting Update request from peer 54534138f6814b809a25c03a46e5f931 for earlier term 4. Current term is 5. Ops: []
I20240503 08:13:42.591969 10161 catalog_manager.cc:5484] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 3 to 4, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 4 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.592274 10698 consensus_queue.cc:260] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.592572 10698 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.593482 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab9203ab3c96465682075b911880b320"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.593665 10440 raft_consensus.cc:482] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.593776 10440 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.595396 10566 raft_consensus.cc:1261] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240503 08:13:42.596204 10161 catalog_manager.cc:5484] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.596726 10440 raft_consensus.cc:504] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.598003 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.598278 10700 raft_consensus.cc:3032] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.598439 10700 raft_consensus.cc:729] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.598803 10700 consensus_queue.cc:260] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.599246 10700 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.599961 12414 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:42.600234 10698 raft_consensus.cc:2396] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.602442 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.602687 10305 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.605360 10305 raft_consensus.cc:2454] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.605880 10374 leader_election.cc:302] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 54534138f6814b809a25c03a46e5f931
I20240503 08:13:42.606436 12708 raft_consensus.cc:2781] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.606738 12708 raft_consensus.cc:686] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.606688 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.606920 10567 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.607144 12708 consensus_queue.cc:237] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.610369 10567 raft_consensus.cc:2396] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:13:42.611491 10440 leader_election.cc:288] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.612915 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.613195 10440 raft_consensus.cc:482] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.613310 10440 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.616677 10440 raft_consensus.cc:504] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.618103 10698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.618402 10698 raft_consensus.cc:3032] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.618496 10698 raft_consensus.cc:729] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.618793 10698 consensus_queue.cc:260] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.618980 10439 raft_consensus.cc:1226] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Rejecting Update request from peer 54534138f6814b809a25c03a46e5f931 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20240503 08:13:42.619117 10698 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.619540 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.619843 10571 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.623188 10698 raft_consensus.cc:2396] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:13:42.623790 10440 leader_election.cc:288] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.624949 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.625169 10440 raft_consensus.cc:482] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.625310 10440 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.628096 10440 raft_consensus.cc:504] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.628612 10700 raft_consensus.cc:2396] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:13:42.629266 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.629381 10373 leader_election.cc:302] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71; no voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:42.629510 10567 raft_consensus.cc:3032] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:13:42.629591 10567 raft_consensus.cc:729] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:42.629957 12890 raft_consensus.cc:2726] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 08:13:42.629863 10567 consensus_queue.cc:260] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.630259 10567 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.631244 10566 raft_consensus.cc:1261] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240503 08:13:42.631449 10566 pending_rounds.cc:76] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2: Aborting all ops after (but not including) 4
I20240503 08:13:42.631564 10566 pending_rounds.cc:97] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240503 08:13:42.631752 10566 raft_consensus.cc:2866] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:42.632185 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.632510 10697 raft_consensus.cc:3032] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.632623 10697 raft_consensus.cc:729] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.633003 10697 consensus_queue.cc:260] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.634032 10567 raft_consensus.cc:2454] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 4.
I20240503 08:13:42.634598 10376 leader_election.cc:302] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:42.634955 12890 raft_consensus.cc:2781] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:13:42.635099 12890 raft_consensus.cc:686] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 4 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.635377 10308 raft_consensus.cc:1261] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240503 08:13:42.635401 12890 consensus_queue.cc:237] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.636183 12047 consensus_queue.cc:1035] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:42.637328 10440 leader_election.cc:288] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 4 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:42.637635 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.637928 10308 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:13:42.641933 10308 raft_consensus.cc:2454] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 4.
I20240503 08:13:42.646052 10571 raft_consensus.cc:2396] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 08:13:42.646742 10376 leader_election.cc:302] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71; no voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:42.647446 12780 raft_consensus.cc:2726] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 08:13:42.633432 10697 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.652034 10305 raft_consensus.cc:1261] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240503 08:13:42.649608 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.652731 10440 raft_consensus.cc:482] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.652882 10440 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.659729 10697 raft_consensus.cc:2396] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:42.659998 10440 raft_consensus.cc:504] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.660801 10440 leader_election.cc:288] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.661324 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.661566 10305 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.661973 12865 consensus_queue.cc:1035] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.662133 10440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd"
dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
from {username='slave'} at 127.0.0.1:36006
I20240503 08:13:42.662307 10440 raft_consensus.cc:482] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.662427 10440 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.664243 12866 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:42.663882 10571 raft_consensus.cc:1261] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240503 08:13:42.665343 12866 consensus_queue.cc:1035] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:42.670082 10305 raft_consensus.cc:2454] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 5.
I20240503 08:13:42.670733 10374 leader_election.cc:302] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.671149 12780 raft_consensus.cc:2781] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.671340 12780 raft_consensus.cc:686] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.671608 12780 consensus_queue.cc:237] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.673380 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.673595 10697 raft_consensus.cc:3032] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.673676 10697 raft_consensus.cc:729] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.673936 10697 consensus_queue.cc:260] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.674198 10697 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.676805 10697 raft_consensus.cc:2396] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.690022 10440 raft_consensus.cc:504] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.691570 10309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:42.691825 10309 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.693694 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "2b3a6ff465524577a400de6af7e70f71" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.693949 10697 raft_consensus.cc:3032] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.694041 10697 raft_consensus.cc:729] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.694355 10697 consensus_queue.cc:260] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.694669 10697 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.697103 10309 raft_consensus.cc:2454] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3a6ff465524577a400de6af7e70f71 in term 6.
I20240503 08:13:42.697616 10697 raft_consensus.cc:2396] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b3a6ff465524577a400de6af7e70f71 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:13:42.697567 10374 leader_election.cc:302] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.697896 12890 raft_consensus.cc:2781] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:42.698067 12890 raft_consensus.cc:686] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Becoming Leader. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.698329 12890 consensus_queue.cc:237] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.698516 10440 leader_election.cc:288] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [CANDIDATE]: Term 6 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.704066 10161 catalog_manager.cc:5484] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 5 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 6 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.704770 10161 catalog_manager.cc:5484] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 5 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 6 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.712605 10161 catalog_manager.cc:5484] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.713052 10161 catalog_manager.cc:5484] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.713572 10161 catalog_manager.cc:5484] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 4 to 5, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 5 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.714257 10161 catalog_manager.cc:5484] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 5 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 6 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.716908 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.717226 10309 raft_consensus.cc:482] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.717414 10309 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.714790 10161 catalog_manager.cc:5484] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 reported cstate change: term changed from 3 to 4, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194). New cstate: current_term: 4 leader_uuid: "2b3a6ff465524577a400de6af7e70f71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.720703 10309 raft_consensus.cc:504] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.722514 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.722769 10571 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.722805 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.723155 10697 raft_consensus.cc:3032] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.723280 10697 raft_consensus.cc:729] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.723623 10697 consensus_queue.cc:260] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.724112 10697 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.725996 10571 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:42.726332 10309 leader_election.cc:288] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.726608 10245 leader_election.cc:302] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.727017 12797 raft_consensus.cc:2781] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:42.727210 12797 raft_consensus.cc:686] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.727597 12797 consensus_queue.cc:237] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.731464 10697 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:42.731726 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.731251 10161 catalog_manager.cc:5484] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 5 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 6 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.731966 10309 raft_consensus.cc:482] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.732111 10309 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.738324 10309 raft_consensus.cc:504] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.739995 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.740296 10440 raft_consensus.cc:3032] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.740420 10440 raft_consensus.cc:729] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.740797 10440 consensus_queue.cc:260] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.741182 10440 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.745781 10440 raft_consensus.cc:2396] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.753659 10309 leader_election.cc:288] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.754056 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.754329 10571 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.755836 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.756469 10309 raft_consensus.cc:482] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.756608 10309 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.759299 10571 raft_consensus.cc:2396] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.759815 10245 leader_election.cc:302] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:42.760263 12797 raft_consensus.cc:2726] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:13:42.761315 12345 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:42.759267 10309 raft_consensus.cc:504] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.762599 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.762857 10697 raft_consensus.cc:3032] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.762948 10697 raft_consensus.cc:729] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.763283 10697 consensus_queue.cc:260] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.763736 10697 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.763424 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.764139 10571 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.766795 10697 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:42.767570 10549 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:42.767967 10242 leader_election.cc:302] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.768325 12797 raft_consensus.cc:2781] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:42.768570 12797 raft_consensus.cc:686] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.768991 10309 leader_election.cc:288] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.768931 12797 consensus_queue.cc:237] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.770118 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4764c2955533410188d40d9465156bcc"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.770359 10305 raft_consensus.cc:482] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.770493 10305 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.771207 10309 raft_consensus.cc:1261] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240503 08:13:42.771816 12047 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:42.773365 10305 raft_consensus.cc:504] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.773241 10161 catalog_manager.cc:5484] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 5 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 6 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.773988 10305 leader_election.cc:288] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.774484 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.774696 10697 raft_consensus.cc:3032] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.774784 10697 raft_consensus.cc:729] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.775061 10697 consensus_queue.cc:260] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.775738 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "205f68b8cf6d4b948576765252160224"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.775843 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.775934 10308 raft_consensus.cc:482] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.776022 10697 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.776062 10308 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:42.776072 10566 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.778856 10697 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:42.778864 10308 raft_consensus.cc:504] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.779646 10242 leader_election.cc:302] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.780393 10309 raft_consensus.cc:1261] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:13:42.781282 12866 consensus_queue.cc:1035] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:42.782282 12005 raft_consensus.cc:2781] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:42.782470 12005 raft_consensus.cc:686] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.782878 10567 raft_consensus.cc:1261] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240503 08:13:42.782809 12005 consensus_queue.cc:237] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.783007 10567 pending_rounds.cc:76] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2: Aborting all ops after (but not including) 5
I20240503 08:13:42.783100 10567 pending_rounds.cc:97] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240503 08:13:42.783186 10567 raft_consensus.cc:2866] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:42.783946 12865 consensus_queue.cc:1035] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:42.785049 10567 raft_consensus.cc:1261] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240503 08:13:42.786159 10308 leader_election.cc:288] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 7 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.786464 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.786700 10697 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:42.787148 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f68b8cf6d4b948576765252160224" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.787426 10440 raft_consensus.cc:3032] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:42.787524 10440 raft_consensus.cc:729] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.787824 10440 consensus_queue.cc:260] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.788105 10440 raft_consensus.cc:3037] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:42.789741 10697 raft_consensus.cc:2396] T 205f68b8cf6d4b948576765252160224 P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:42.791188 12865 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:42.792210 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.792400 10309 raft_consensus.cc:482] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.792521 10309 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.795110 10309 raft_consensus.cc:504] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.796209 10308 raft_consensus.cc:1261] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:13:42.797092 12865 consensus_queue.cc:1035] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:42.799363 10440 raft_consensus.cc:2396] T 205f68b8cf6d4b948576765252160224 P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:42.800334 10566 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:42.801157 10245 leader_election.cc:302] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:42.801893 10309 leader_election.cc:288] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.803575 10566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.803934 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.805047 10566 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.805332 10440 raft_consensus.cc:3032] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.805609 12005 raft_consensus.cc:2726] T 205f68b8cf6d4b948576765252160224 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:13:42.809652 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9f91eb635604d20853c9f7518164789"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.809912 10309 raft_consensus.cc:482] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.810042 10309 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.810457 12608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:42.810464 10440 raft_consensus.cc:729] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.810945 10440 consensus_queue.cc:260] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.811420 10440 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.812955 10309 raft_consensus.cc:504] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.813669 10309 leader_election.cc:288] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.814926 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.815151 10439 raft_consensus.cc:3032] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.815230 10439 raft_consensus.cc:729] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.815573 10439 consensus_queue.cc:260] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.815894 10439 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.816255 10161 catalog_manager.cc:5484] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 5 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 6 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.820149 10309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d01df9a11104a33a7971e133310acce"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.820382 10439 raft_consensus.cc:2396] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:13:42.820454 10309 raft_consensus.cc:482] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.820672 10309 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.824019 10309 raft_consensus.cc:504] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.824455 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.824846 10567 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.825382 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.825505 10570 raft_consensus.cc:1261] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer 54534138f6814b809a25c03a46e5f931: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240503 08:13:42.825570 10569 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.825663 10570 pending_rounds.cc:76] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2: Aborting all ops after (but not including) 5
I20240503 08:13:42.825733 10570 pending_rounds.cc:97] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240503 08:13:42.825830 10570 raft_consensus.cc:2866] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:42.826449 12866 consensus_queue.cc:1035] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:42.829520 10569 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:42.830044 10245 leader_election.cc:302] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.830504 12797 raft_consensus.cc:2781] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:42.830646 12797 raft_consensus.cc:686] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.830936 12797 consensus_queue.cc:237] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.833249 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.833479 10697 raft_consensus.cc:3032] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.833521 10309 leader_election.cc:288] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.833601 10697 raft_consensus.cc:729] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.833873 10697 consensus_queue.cc:260] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.834501 10697 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.834928 10567 raft_consensus.cc:2454] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:42.837529 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57310c72619e44c99fa7f42673648658"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.837774 10305 raft_consensus.cc:482] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.837903 10305 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.838757 10245 leader_election.cc:302] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:42.839221 12005 raft_consensus.cc:2781] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:42.839406 12005 raft_consensus.cc:686] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.839730 12005 consensus_queue.cc:237] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.840818 10305 raft_consensus.cc:504] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.841717 10440 raft_consensus.cc:2396] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:13:42.841939 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.842021 10571 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:42.842087 10305 leader_election.cc:288] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 5 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.842167 10439 raft_consensus.cc:3032] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.842244 10439 raft_consensus.cc:729] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.842553 10439 consensus_queue.cc:260] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.842892 10439 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.837029 10566 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:42.843964 10245 leader_election.cc:302] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:42.845019 12797 raft_consensus.cc:2781] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:42.845188 12797 raft_consensus.cc:686] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.845445 12797 consensus_queue.cc:237] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.848191 10439 raft_consensus.cc:2396] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:42.848508 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.848752 10305 raft_consensus.cc:482] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.848877 10305 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.851852 10305 raft_consensus.cc:504] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.851953 10168 catalog_manager.cc:5484] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 5 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 6 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.852950 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.853178 10439 raft_consensus.cc:3032] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.853274 10439 raft_consensus.cc:729] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.853557 10439 consensus_queue.cc:260] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.853868 10439 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.857062 10439 raft_consensus.cc:2396] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:42.860431 10168 catalog_manager.cc:5484] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 5 to 6, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 6 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.861434 10168 catalog_manager.cc:5484] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 5 to 6, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 6 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.863793 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.864168 10697 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:42.864442 10567 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.868116 10305 leader_election.cc:288] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 5 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:42.868187 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.868402 10568 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.871311 10568 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 5.
I20240503 08:13:42.872623 10245 leader_election.cc:302] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:42.873122 12005 raft_consensus.cc:2781] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.873279 12005 raft_consensus.cc:686] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.873560 12005 consensus_queue.cc:237] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.875265 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8127b76db73402d8100faad68271644"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.875576 10305 raft_consensus.cc:482] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.875775 10305 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.878733 10305 raft_consensus.cc:504] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.883427 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:42.883765 10568 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.886737 10568 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 5.
I20240503 08:13:42.887521 10245 leader_election.cc:302] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:42.887990 12797 raft_consensus.cc:2781] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.888432 12797 raft_consensus.cc:686] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.888774 12797 consensus_queue.cc:237] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.891438 10305 leader_election.cc:288] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 5 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.893275 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.893878 10567 raft_consensus.cc:2454] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 5.
I20240503 08:13:42.894960 10245 leader_election.cc:302] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:42.895525 12797 raft_consensus.cc:2781] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:42.895687 12797 raft_consensus.cc:686] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:42.895964 12797 consensus_queue.cc:237] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:42.899526 10168 catalog_manager.cc:5484] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 4 to 5, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 5 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.902211 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.902518 10697 raft_consensus.cc:3032] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:42.902704 10697 raft_consensus.cc:729] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:42.902901 10305 raft_consensus.cc:482] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.903265 10697 consensus_queue.cc:260] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.904398 10568 raft_consensus.cc:1261] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240503 08:13:42.903735 10305 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.905508 12708 consensus_queue.cc:1035] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:42.908371 10305 raft_consensus.cc:504] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.909091 10697 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:42.908440 10168 catalog_manager.cc:5484] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 4 to 5, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 5 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:42.911276 10168 catalog_manager.cc:5484] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 4 to 5, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 5 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:42.911990 10418 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 08:13:42.912231 10697 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 5.
I20240503 08:13:42.913617 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.913842 10700 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.916885 10700 raft_consensus.cc:2396] T 9d3c45457bd446fb8dec0197c8e809ce P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.918967 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3c45457bd446fb8dec0197c8e809ce" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:42.919258 10440 raft_consensus.cc:3032] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.919376 10440 raft_consensus.cc:729] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.919710 10440 consensus_queue.cc:260] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.920042 10440 raft_consensus.cc:3037] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.904853 10567 raft_consensus.cc:1261] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240503 08:13:42.923352 12890 consensus_queue.cc:1035] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:42.925462 10700 raft_consensus.cc:1261] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20240503 08:13:42.926337 10440 raft_consensus.cc:2396] T 9d3c45457bd446fb8dec0197c8e809ce P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.926517 10700 pending_rounds.cc:76] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 4
I20240503 08:13:42.926647 10700 pending_rounds.cc:97] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240503 08:13:42.926801 10700 raft_consensus.cc:2866] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:42.927771 12780 consensus_queue.cc:1035] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.928244 10245 leader_election.cc:302] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:42.928628 12005 raft_consensus.cc:2726] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:13:42.928833 10308 raft_consensus.cc:1261] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240503 08:13:42.929639 12708 consensus_queue.cc:1035] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:42.937563 10305 leader_election.cc:288] T 9d3c45457bd446fb8dec0197c8e809ce P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.938778 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.939000 10305 raft_consensus.cc:482] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.939101 10305 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.942009 10305 raft_consensus.cc:504] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.944811 10701 raft_consensus.cc:1261] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20240503 08:13:42.944867 10308 raft_consensus.cc:1226] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Rejecting Update request from peer 2b3a6ff465524577a400de6af7e70f71 for earlier term 5. Current term is 6. Ops: []
I20240503 08:13:42.944927 10701 pending_rounds.cc:76] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 4
I20240503 08:13:42.944995 10701 pending_rounds.cc:97] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240503 08:13:42.945055 10701 raft_consensus.cc:2866] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:42.945569 12780 consensus_queue.cc:1046] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.945849 12891 raft_consensus.cc:3032] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:42.945976 12891 raft_consensus.cc:729] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:42.946216 12891 consensus_queue.cc:260] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:42.946527 12891 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.948062 12780 consensus_queue.cc:1035] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 08:13:42.948676 10697 raft_consensus.cc:1261] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20240503 08:13:42.948781 10697 pending_rounds.cc:76] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 5
I20240503 08:13:42.948848 10697 pending_rounds.cc:97] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20240503 08:13:42.948904 10697 raft_consensus.cc:2866] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:42.949970 12780 consensus_queue.cc:1035] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:42.950963 10308 raft_consensus.cc:1261] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:13:42.955660 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.955950 10700 raft_consensus.cc:3037] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.958990 10700 raft_consensus.cc:2396] T 43fedbb54bc34fa49f69d4891323405b P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:42.965523 10305 leader_election.cc:288] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:42.969813 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.970516 10305 raft_consensus.cc:482] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.970877 10305 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:42.975147 10305 raft_consensus.cc:504] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:42.979912 10305 leader_election.cc:288] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 7 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.981743 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3715b584822e4cd3bee05119936125d0"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.982071 10305 raft_consensus.cc:482] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.982347 10305 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.986363 10305 raft_consensus.cc:504] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:42.989171 10305 leader_election.cc:288] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:42.989324 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.990646 10697 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:42.990947 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:42.991170 10700 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:42.995591 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:42.995885 10305 raft_consensus.cc:482] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:42.996076 10305 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.000141 10305 raft_consensus.cc:504] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.001719 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:43.001991 10568 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.005009 10305 leader_election.cc:288] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:43.005018 10568 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:43.005477 10245 leader_election.cc:302] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:43.005808 12005 raft_consensus.cc:2781] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:43.006001 12005 raft_consensus.cc:686] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.006268 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:43.006361 10700 raft_consensus.cc:2396] T 3715b584822e4cd3bee05119936125d0 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:43.006307 12005 consensus_queue.cc:237] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.006489 10305 raft_consensus.cc:482] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.006609 10305 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:43.010031 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.010447 10700 raft_consensus.cc:3032] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.010627 10700 raft_consensus.cc:729] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:43.010527 10161 catalog_manager.cc:5484] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 5 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 6 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:43.018088 10305 raft_consensus.cc:504] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.011242 10700 consensus_queue.cc:260] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.019449 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:43.019690 10568 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:43.019673 10700 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.023227 10700 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:43.024889 10568 raft_consensus.cc:2454] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 5.
I20240503 08:13:43.025539 10245 leader_election.cc:302] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:43.025894 12797 raft_consensus.cc:2781] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:13:43.026067 12797 raft_consensus.cc:686] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.026348 12797 consensus_queue.cc:237] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.029189 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43fedbb54bc34fa49f69d4891323405b" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.029893 10161 catalog_manager.cc:5484] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 4 to 5, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 5 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.030740 10440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c636ced06a44624910078f2e75eea2f" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
W20240503 08:13:43.031872 10373 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.196:36257, user_credentials={real_user=slave}} blocked reactor thread for 72170us
I20240503 08:13:43.027245 12780 consensus_queue.cc:1035] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20240503 08:13:43.035029 10567 raft_consensus.cc:1261] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:13:43.038300 10697 raft_consensus.cc:2396] T 6c636ced06a44624910078f2e75eea2f P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:43.040071 10440 raft_consensus.cc:3032] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.040205 10440 raft_consensus.cc:729] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.040511 10440 consensus_queue.cc:260] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.040829 10440 raft_consensus.cc:3037] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.042788 10679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 08:13:43.042903 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3715b584822e4cd3bee05119936125d0" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.043150 10438 raft_consensus.cc:3032] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.043247 10438 raft_consensus.cc:729] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.043519 10438 consensus_queue.cc:260] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.043875 10438 raft_consensus.cc:3037] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.043992 10440 raft_consensus.cc:2396] T 6c636ced06a44624910078f2e75eea2f P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 08:13:43.044812 10305 leader_election.cc:288] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 5 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:43.046001 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f8e6483973f420e97503f409130cc36"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:43.046202 10305 raft_consensus.cc:482] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.046352 10305 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.046348 12476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:43.046512 10438 raft_consensus.cc:2396] T 3715b584822e4cd3bee05119936125d0 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:43.048960 10700 raft_consensus.cc:1261] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240503 08:13:43.049088 10700 pending_rounds.cc:76] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 5
I20240503 08:13:43.049165 10700 pending_rounds.cc:97] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240503 08:13:43.049233 10700 raft_consensus.cc:2866] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:43.049492 10305 raft_consensus.cc:504] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.050908 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:43.051154 10568 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.052111 12213 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 08:13:43.053135 12708 consensus_queue.cc:1035] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20240503 08:13:43.053633 10700 raft_consensus.cc:1261] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240503 08:13:43.053750 10700 pending_rounds.cc:76] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 5
I20240503 08:13:43.053834 10700 pending_rounds.cc:97] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240503 08:13:43.053913 10700 raft_consensus.cc:2866] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:43.054229 10568 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:43.054786 10287 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 08:13:43.055364 10245 leader_election.cc:302] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:43.055729 12005 raft_consensus.cc:2781] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:43.055871 12005 raft_consensus.cc:686] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.056149 12005 consensus_queue.cc:237] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.060504 10305 leader_election.cc:288] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:43.060699 12891 consensus_queue.cc:1035] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.061784 10439 raft_consensus.cc:2396] T 43fedbb54bc34fa49f69d4891323405b P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:43.061594 12780 consensus_queue.cc:1035] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20240503 08:13:43.079927 10305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:43.080322 10305 raft_consensus.cc:482] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.080507 10305 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.080727 10161 catalog_manager.cc:5484] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 5 to 6, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 6 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.091006 10308 raft_consensus.cc:1226] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Rejecting Update request from peer 2b3a6ff465524577a400de6af7e70f71 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20240503 08:13:43.091192 10700 raft_consensus.cc:1261] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20240503 08:13:43.091311 10700 pending_rounds.cc:76] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 5
I20240503 08:13:43.091418 10700 pending_rounds.cc:97] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20240503 08:13:43.091500 10700 raft_consensus.cc:2866] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:43.089862 10305 raft_consensus.cc:504] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.092247 12780 consensus_queue.cc:1035] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.092490 10308 raft_consensus.cc:1226] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Rejecting Update request from peer 2b3a6ff465524577a400de6af7e70f71 for earlier term 5. Current term is 6. Ops: []
I20240503 08:13:43.092850 10305 leader_election.cc:288] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:43.093461 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:43.093699 10568 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.093914 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:43.094101 10308 raft_consensus.cc:482] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.094201 10308 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.096712 10568 raft_consensus.cc:2396] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:43.097800 10308 raft_consensus.cc:504] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.098974 10308 leader_election.cc:288] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:43.099347 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:43.099656 10568 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.100086 12929 consensus_queue.cc:1046] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.100445 12929 raft_consensus.cc:3032] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.101085 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:43.101387 10308 raft_consensus.cc:482] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.101579 10308 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.102571 10568 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:43.102748 12929 raft_consensus.cc:729] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.103057 10697 raft_consensus.cc:1261] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240503 08:13:43.103098 10245 leader_election.cc:302] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:43.103201 10697 pending_rounds.cc:76] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931: Aborting all ops after (but not including) 5
I20240503 08:13:43.103260 10697 pending_rounds.cc:97] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240503 08:13:43.103395 10697 raft_consensus.cc:2866] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:43.103478 12797 raft_consensus.cc:2781] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:43.103644 12797 raft_consensus.cc:686] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Leader. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.103935 12797 consensus_queue.cc:237] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.104480 12929 consensus_queue.cc:260] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.107609 10168 catalog_manager.cc:5484] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a reported cstate change: term changed from 5 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193). New cstate: current_term: 6 leader_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.112007 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.112298 10697 raft_consensus.cc:3032] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.112391 10697 raft_consensus.cc:729] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:43.112704 10697 consensus_queue.cc:260] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.113116 10697 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.113745 12929 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.115803 12780 consensus_queue.cc:1035] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.118211 10697 raft_consensus.cc:2454] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:43.119249 10308 raft_consensus.cc:504] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.120865 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.121133 10700 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
W20240503 08:13:43.131981 10376 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.157.195:44929, user_credentials={real_user=slave}} blocked reactor thread for 94841.1us
I20240503 08:13:43.132937 10245 leader_election.cc:302] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:43.133561 10245 leader_election.cc:302] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:43.133896 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.134016 10245 leader_election.cc:302] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:43.134132 10438 raft_consensus.cc:3032] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.134249 10438 raft_consensus.cc:729] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.134517 10438 consensus_queue.cc:260] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.134925 10438 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.135483 10567 raft_consensus.cc:1226] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Rejecting Update request from peer 2b3a6ff465524577a400de6af7e70f71 for earlier term 5. Current term is 6. Ops: []
I20240503 08:13:43.135527 12005 raft_consensus.cc:2726] T 6c636ced06a44624910078f2e75eea2f P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:13:43.135682 12005 raft_consensus.cc:2726] T 3715b584822e4cd3bee05119936125d0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:13:43.135802 12005 raft_consensus.cc:2726] T 43fedbb54bc34fa49f69d4891323405b P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:13:43.136075 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.136293 10439 raft_consensus.cc:3032] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:13:43.136385 10439 raft_consensus.cc:729] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.136662 10439 consensus_queue.cc:260] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.136981 10439 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:13:43.138329 10438 raft_consensus.cc:2454] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d2dd10f0d54ac08dfdb8d2a348044a in term 6.
I20240503 08:13:43.139011 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.139256 10438 raft_consensus.cc:2396] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:43.139703 10566 raft_consensus.cc:1261] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240503 08:13:43.139714 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "471f8f96b28a4af2b70604a7fd1628fe" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.139860 10438 raft_consensus.cc:3032] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.139961 10438 raft_consensus.cc:729] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.140252 10571 raft_consensus.cc:1261] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:13:43.140213 10438 consensus_queue.cc:260] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.140527 10438 raft_consensus.cc:3037] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.141662 10439 raft_consensus.cc:2396] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:43.142146 10568 raft_consensus.cc:1261] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer 2b3a6ff465524577a400de6af7e70f71: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:13:43.142886 10700 raft_consensus.cc:2396] T 471f8f96b28a4af2b70604a7fd1628fe P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:43.143502 10438 raft_consensus.cc:2396] T 471f8f96b28a4af2b70604a7fd1628fe P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 08:13:43.144244 10245 leader_election.cc:302] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:43.144384 12708 consensus_queue.cc:1035] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.145126 12919 raft_consensus.cc:2726] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:13:43.145239 12708 consensus_queue.cc:1035] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.145998 12708 consensus_queue.cc:1035] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.147727 10242 leader_election.cc:302] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:43.148064 12919 raft_consensus.cc:2726] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:13:43.131459 10308 leader_election.cc:288] T 471f8f96b28a4af2b70604a7fd1628fe P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 6 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:43.156311 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd"
dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
from {username='slave'} at 127.0.0.1:57024
I20240503 08:13:43.156653 10308 raft_consensus.cc:482] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.156833 10308 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.161839 10308 raft_consensus.cc:504] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.163599 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.163820 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965d21fd6ae94abf96f96e224ccb7bdd" candidate_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.163887 10700 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.164022 10438 raft_consensus.cc:3032] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.164113 10438 raft_consensus.cc:729] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.164372 10438 consensus_queue.cc:260] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.164667 10438 raft_consensus.cc:3037] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.167140 10700 raft_consensus.cc:2396] T 965d21fd6ae94abf96f96e224ccb7bdd P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:13:43.167819 10438 raft_consensus.cc:2396] T 965d21fd6ae94abf96f96e224ccb7bdd P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7d2dd10f0d54ac08dfdb8d2a348044a for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:13:43.178757 10308 leader_election.cc:288] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 7 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:43.179306 10245 leader_election.cc:302] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931
I20240503 08:13:43.179735 12919 raft_consensus.cc:2726] T 965d21fd6ae94abf96f96e224ccb7bdd P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:13:43.180565 10569 raft_consensus.cc:1261] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240503 08:13:43.181396 12919 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:43.186671 10700 raft_consensus.cc:1261] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240503 08:13:43.187376 12005 consensus_queue.cc:1035] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:43.195641 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.195891 10569 raft_consensus.cc:482] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.196017 10569 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.201400 10569 raft_consensus.cc:504] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.202747 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.202880 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.202997 10700 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.203151 10438 raft_consensus.cc:3032] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.203285 10438 raft_consensus.cc:729] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.203702 10438 consensus_queue.cc:260] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.204218 10438 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.206089 10700 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.206774 10504 leader_election.cc:302] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:43.207124 12852 raft_consensus.cc:2781] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:43.207372 12852 raft_consensus.cc:686] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 6 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.207391 10438 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.207545 10569 leader_election.cc:288] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:43.207772 12852 consensus_queue.cc:237] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.208858 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.209101 10569 raft_consensus.cc:482] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.209221 10569 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.212867 10700 raft_consensus.cc:1261] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240503 08:13:43.212826 10569 raft_consensus.cc:504] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.213449 12005 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:43.213593 10569 leader_election.cc:288] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:43.214846 10701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.215062 10701 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.215032 10161 catalog_manager.cc:5484] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 5 to 6, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 6 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.218027 10701 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.221161 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.221464 10569 raft_consensus.cc:482] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.221634 10569 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.221639 10571 raft_consensus.cc:1261] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240503 08:13:43.224838 10569 raft_consensus.cc:504] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.221233 10504 leader_election.cc:302] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:43.226310 12785 raft_consensus.cc:2781] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.226542 12785 raft_consensus.cc:686] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.226743 10569 leader_election.cc:288] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:43.226828 12785 consensus_queue.cc:237] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.227311 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.227465 12005 consensus_queue.cc:1035] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:43.227595 10308 raft_consensus.cc:3032] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.227715 10308 raft_consensus.cc:729] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.227977 10308 consensus_queue.cc:260] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.228304 10308 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.230175 10569 raft_consensus.cc:1261] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240503 08:13:43.230723 12919 consensus_queue.cc:1035] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.232295 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4670994e75a24234913724bac4c30ec4"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.232522 10571 raft_consensus.cc:482] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.232674 10571 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.232266 10162 catalog_manager.cc:5484] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 6 to 7, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 7 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:43.235348 10308 raft_consensus.cc:2396] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240503 08:13:43.235778 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.235908 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.236166 10700 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.236419 10438 raft_consensus.cc:3032] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.236478 10571 raft_consensus.cc:504] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.237377 10438 raft_consensus.cc:729] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.239457 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.237776 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.239728 10697 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.239840 10439 raft_consensus.cc:3032] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.239930 10439 raft_consensus.cc:729] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.240010 10700 raft_consensus.cc:2454] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.240250 10439 consensus_queue.cc:260] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.240717 10439 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.241081 10571 leader_election.cc:288] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:43.241885 10438 consensus_queue.cc:260] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.242717 10697 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.243026 10438 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.243213 10504 leader_election.cc:302] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters: d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:43.243880 12852 raft_consensus.cc:2781] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.244047 12852 raft_consensus.cc:686] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.244411 12852 consensus_queue.cc:237] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.246098 10504 leader_election.cc:302] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:43.246426 12785 raft_consensus.cc:2781] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:43.246603 12785 raft_consensus.cc:686] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [term 6 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.246905 12785 consensus_queue.cc:237] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.247061 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.247275 10571 raft_consensus.cc:482] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.247463 10571 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.249455 10439 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.251204 10571 raft_consensus.cc:504] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.252744 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.253017 10308 raft_consensus.cc:3032] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.253137 10308 raft_consensus.cc:729] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.253454 10308 consensus_queue.cc:260] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.253809 10308 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.256058 10438 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.258445 10308 raft_consensus.cc:2396] T d44e068b52864dde9d988775249cb8d2 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240503 08:13:43.259126 10571 leader_election.cc:288] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:43.259752 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.260025 10697 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.260836 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4764c2955533410188d40d9465156bcc"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.261147 10571 raft_consensus.cc:482] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.261365 10571 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.263100 10697 raft_consensus.cc:2454] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.263679 10504 leader_election.cc:302] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters: d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:43.264189 12785 raft_consensus.cc:2781] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.264364 12785 raft_consensus.cc:686] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.264654 12785 consensus_queue.cc:237] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.264950 10571 raft_consensus.cc:504] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.266556 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.266810 10308 raft_consensus.cc:3032] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.266939 10308 raft_consensus.cc:729] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.267131 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764c2955533410188d40d9465156bcc" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.267429 10697 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.267285 10308 consensus_queue.cc:260] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.267778 10308 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.271082 10697 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.271169 10308 raft_consensus.cc:2454] T 4764c2955533410188d40d9465156bcc P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.271440 10571 leader_election.cc:288] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:43.273329 10438 raft_consensus.cc:1261] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20240503 08:13:43.273443 10438 pending_rounds.cc:76] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71: Aborting all ops after (but not including) 5
I20240503 08:13:43.273589 10438 pending_rounds.cc:97] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20240503 08:13:43.273677 10438 raft_consensus.cc:2866] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:43.274456 12797 consensus_queue.cc:1035] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.276435 10504 leader_election.cc:302] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:43.276145 10161 catalog_manager.cc:5484] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 6 to 7, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 7 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.276873 10161 catalog_manager.cc:5484] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 6 to 7, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 7 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.277462 10161 catalog_manager.cc:5484] T 4670994e75a24234913724bac4c30ec4 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 5 to 6, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 6 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.278717 12785 raft_consensus.cc:2781] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.278887 12785 raft_consensus.cc:686] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.279211 12785 consensus_queue.cc:237] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.283502 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.283723 10569 raft_consensus.cc:482] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.283872 10569 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.286787 10569 raft_consensus.cc:504] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.288103 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.288398 10308 raft_consensus.cc:3032] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.288527 10308 raft_consensus.cc:729] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.288872 10308 consensus_queue.cc:260] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.289115 10168 catalog_manager.cc:5484] T 4764c2955533410188d40d9465156bcc P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 6 to 7, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 7 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.289386 10308 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.291821 10569 leader_election.cc:288] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:43.293164 10308 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.293745 10505 leader_election.cc:302] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters:
I20240503 08:13:43.294021 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835ed4599b904cd695e171afe6e65efa" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.294107 12785 raft_consensus.cc:2781] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.294245 10438 raft_consensus.cc:3037] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.294291 12785 raft_consensus.cc:686] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.294616 12785 consensus_queue.cc:237] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.295116 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9f91eb635604d20853c9f7518164789"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.295364 10571 raft_consensus.cc:482] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.295490 10571 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.297080 10568 raft_consensus.cc:1261] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:13:43.297684 12797 consensus_queue.cc:1035] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.298537 10571 raft_consensus.cc:504] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.299506 10571 leader_election.cc:288] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:43.300087 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.300309 10439 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.300946 10168 catalog_manager.cc:5484] T 835ed4599b904cd695e171afe6e65efa P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 6 to 7, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 7 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.302162 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f91eb635604d20853c9f7518164789" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.302419 10305 raft_consensus.cc:3032] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.302542 10305 raft_consensus.cc:729] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.302858 10305 consensus_queue.cc:260] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.303201 10305 raft_consensus.cc:3037] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.303313 10439 raft_consensus.cc:2396] T f9f91eb635604d20853c9f7518164789 P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:13:43.306468 10305 raft_consensus.cc:2396] T f9f91eb635604d20853c9f7518164789 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:13:43.306996 10505 leader_election.cc:302] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2; no voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:43.307590 12785 raft_consensus.cc:2726] T f9f91eb635604d20853c9f7518164789 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:13:43.307741 10568 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d01df9a11104a33a7971e133310acce"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.307972 10568 raft_consensus.cc:482] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.308095 10568 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.309895 10697 raft_consensus.cc:1261] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:13:43.310629 12919 consensus_queue.cc:1035] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.311161 10568 raft_consensus.cc:504] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.314707 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.314998 10305 raft_consensus.cc:3032] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.315099 10305 raft_consensus.cc:729] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.315424 10568 leader_election.cc:288] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:43.315929 10305 consensus_queue.cc:260] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.316542 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d01df9a11104a33a7971e133310acce" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.316677 10305 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.316746 10700 raft_consensus.cc:3037] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.317750 10571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.317862 10566 raft_consensus.cc:1261] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 08:13:43.317953 10571 raft_consensus.cc:482] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.318081 10571 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.319595 10700 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.320616 10504 leader_election.cc:302] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:43.320938 12785 raft_consensus.cc:2781] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.321110 12785 raft_consensus.cc:686] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.321174 12919 consensus_queue.cc:1035] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.321390 12785 consensus_queue.cc:237] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.326525 10438 raft_consensus.cc:2454] T 835ed4599b904cd695e171afe6e65efa P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.327479 10571 raft_consensus.cc:504] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.329027 10305 raft_consensus.cc:2454] T 7d01df9a11104a33a7971e133310acce P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.331909 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.332263 10438 raft_consensus.cc:3032] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.332388 10438 raft_consensus.cc:729] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.332826 10438 consensus_queue.cc:260] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.333482 10438 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.337757 10438 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.338500 10507 leader_election.cc:302] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:43.338954 12852 raft_consensus.cc:2781] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.339160 12852 raft_consensus.cc:686] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.339562 12852 consensus_queue.cc:237] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.342981 10571 leader_election.cc:288] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:43.343011 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a424b97c9b453081b568ab96bead7a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.343278 10700 raft_consensus.cc:3037] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.344296 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57310c72619e44c99fa7f42673648658"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.344585 10569 raft_consensus.cc:482] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.344717 10569 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.346190 10700 raft_consensus.cc:2454] T a3a424b97c9b453081b568ab96bead7a P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.349162 10569 raft_consensus.cc:504] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.351485 10569 leader_election.cc:288] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:43.353058 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.353654 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.354302 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57310c72619e44c99fa7f42673648658" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.354552 10438 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.353941 10305 raft_consensus.cc:3032] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.355652 10305 raft_consensus.cc:729] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.356094 10305 consensus_queue.cc:260] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.356550 10305 raft_consensus.cc:3037] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.358335 10439 raft_consensus.cc:1261] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240503 08:13:43.358448 10439 pending_rounds.cc:76] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71: Aborting all ops after (but not including) 5
I20240503 08:13:43.358507 10439 pending_rounds.cc:97] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240503 08:13:43.358562 10439 raft_consensus.cc:2866] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:13:43.359341 12919 consensus_queue.cc:1035] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.364678 10438 raft_consensus.cc:2396] T 57310c72619e44c99fa7f42673648658 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:43.366727 10305 raft_consensus.cc:2396] T 57310c72619e44c99fa7f42673648658 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:43.367231 10569 raft_consensus.cc:482] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.367426 10569 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.368126 10505 leader_election.cc:302] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2; no voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:43.368618 12852 raft_consensus.cc:2726] T 57310c72619e44c99fa7f42673648658 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:13:43.370355 10569 raft_consensus.cc:504] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.370280 10168 catalog_manager.cc:5484] T 7d01df9a11104a33a7971e133310acce P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 6 to 7, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 7 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.370760 10568 raft_consensus.cc:1226] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Rejecting Update request from peer d7d2dd10f0d54ac08dfdb8d2a348044a for earlier term 5. Current term is 6. Ops: []
I20240503 08:13:43.371402 10569 leader_election.cc:288] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:43.371716 12919 consensus_queue.cc:1046] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 08:13:43.371979 12919 raft_consensus.cc:3032] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.372074 12919 raft_consensus.cc:729] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.372345 12919 consensus_queue.cc:260] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.372689 12919 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.372800 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8127b76db73402d8100faad68271644"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.372989 10569 raft_consensus.cc:482] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.373090 10569 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.373675 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.373875 10439 raft_consensus.cc:3037] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.375958 10569 raft_consensus.cc:504] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.376739 10439 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.377234 10507 leader_election.cc:302] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:43.377558 12852 raft_consensus.cc:2781] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:43.377573 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c422532250a4ba3bf1553722462b9da" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.377713 12852 raft_consensus.cc:686] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [term 6 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.377948 12852 consensus_queue.cc:237] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.378782 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.379024 10700 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.381592 10700 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.382261 10569 leader_election.cc:288] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:43.382279 10504 leader_election.cc:302] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:43.383311 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8127b76db73402d8100faad68271644" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.383575 10308 raft_consensus.cc:3032] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.383682 10308 raft_consensus.cc:729] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.383996 10308 consensus_queue.cc:260] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.384438 10308 raft_consensus.cc:3037] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.385558 10305 raft_consensus.cc:2454] T 3c422532250a4ba3bf1553722462b9da P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.387044 12852 raft_consensus.cc:2781] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:43.387228 12852 raft_consensus.cc:686] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [term 6 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.387275 10308 raft_consensus.cc:2454] T e8127b76db73402d8100faad68271644 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.387584 12852 consensus_queue.cc:237] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.388913 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab9203ab3c96465682075b911880b320"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.389112 10569 raft_consensus.cc:482] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.389258 10569 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.391974 10168 catalog_manager.cc:5484] T e8127b76db73402d8100faad68271644 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 5 to 6, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 6 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.392282 10569 raft_consensus.cc:504] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.392649 10168 catalog_manager.cc:5484] T a3a424b97c9b453081b568ab96bead7a P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 6 to 7, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 7 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.393088 10569 leader_election.cc:288] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:43.393203 10168 catalog_manager.cc:5484] T 3c422532250a4ba3bf1553722462b9da P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 5 to 6, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 6 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:43.394959 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.395202 10700 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.397979 10700 raft_consensus.cc:2396] T ab9203ab3c96465682075b911880b320 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:43.399629 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9203ab3c96465682075b911880b320" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.399881 10439 raft_consensus.cc:3037] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.402241 10439 raft_consensus.cc:2454] T ab9203ab3c96465682075b911880b320 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.403317 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.403501 10507 leader_election.cc:302] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters: 54534138f6814b809a25c03a46e5f931
I20240503 08:13:43.403651 10569 raft_consensus.cc:482] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.403789 10569 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.404367 12852 raft_consensus.cc:2781] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:43.404551 12852 raft_consensus.cc:686] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [term 6 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.404819 12852 consensus_queue.cc:237] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.407095 10569 raft_consensus.cc:504] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.407821 10569 leader_election.cc:288] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:43.408737 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.408948 10700 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.410972 10700 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.411655 10504 leader_election.cc:302] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:43.411949 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.412096 12785 raft_consensus.cc:2781] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:43.412200 10569 raft_consensus.cc:482] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.412319 10569 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.413025 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6443a176ba74f649ecc8da8717ed879" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.413249 10439 raft_consensus.cc:3032] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.413349 10439 raft_consensus.cc:729] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2b3a6ff465524577a400de6af7e70f71, State: Running, Role: LEADER
I20240503 08:13:43.413628 10439 consensus_queue.cc:260] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.414054 10439 raft_consensus.cc:3037] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.415439 10569 raft_consensus.cc:504] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.416589 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.416853 10700 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.418422 10439 raft_consensus.cc:2454] T f6443a176ba74f649ecc8da8717ed879 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.420486 10700 raft_consensus.cc:2454] T 22b825c0acaa453f837aa7ca7a5543b0 P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.421063 10504 leader_election.cc:302] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:43.427129 10569 leader_election.cc:288] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:43.427250 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b825c0acaa453f837aa7ca7a5543b0" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.427521 10308 raft_consensus.cc:3032] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.427657 10308 raft_consensus.cc:729] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.428004 10308 consensus_queue.cc:260] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.428444 10308 raft_consensus.cc:3037] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.432647 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.432922 10569 raft_consensus.cc:482] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.433051 10569 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.433674 10308 raft_consensus.cc:2396] T 22b825c0acaa453f837aa7ca7a5543b0 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240503 08:13:43.440790 10569 raft_consensus.cc:504] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.441717 10569 leader_election.cc:288] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:43.441895 10439 raft_consensus.cc:1261] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240503 08:13:43.442174 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.442389 10438 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.442714 12919 consensus_queue.cc:1035] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 08:13:43.446094 10438 raft_consensus.cc:2454] T cb392d6697d1442d86d3a64f956e8211 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 6.
I20240503 08:13:43.446980 10507 leader_election.cc:302] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:43.448676 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.449215 10569 raft_consensus.cc:482] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.449548 10569 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.452876 10569 raft_consensus.cc:504] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.459398 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.459846 10308 raft_consensus.cc:3032] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:13:43.459983 10308 raft_consensus.cc:729] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.460247 12947 raft_consensus.cc:2781] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.460479 12947 raft_consensus.cc:686] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.460322 10308 consensus_queue.cc:260] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.460809 10308 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.460866 12947 consensus_queue.cc:237] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.464197 10308 raft_consensus.cc:2396] T 68032d6c4ad949edbf1bb55aea7f5562 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:43.464864 12947 raft_consensus.cc:2781] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:13:43.465094 12947 raft_consensus.cc:686] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [term 6 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.465443 12947 consensus_queue.cc:237] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.465883 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68032d6c4ad949edbf1bb55aea7f5562" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.466178 10439 raft_consensus.cc:3037] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.469219 10439 raft_consensus.cc:2396] T 68032d6c4ad949edbf1bb55aea7f5562 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:43.469959 10507 leader_election.cc:302] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2; no voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:43.471282 12949 raft_consensus.cc:2726] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:13:43.471310 10569 leader_election.cc:288] T 68032d6c4ad949edbf1bb55aea7f5562 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 6 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:43.473011 12785 raft_consensus.cc:686] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [term 6 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.473088 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f8e6483973f420e97503f409130cc36"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.473357 10569 raft_consensus.cc:482] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.473551 12785 consensus_queue.cc:237] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.479396 10569 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.483212 10168 catalog_manager.cc:5484] T ab9203ab3c96465682075b911880b320 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 4 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 6 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.485785 10569 raft_consensus.cc:504] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.486485 10569 leader_election.cc:288] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:43.487572 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.487804 10439 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.487816 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
W20240503 08:13:43.488276 10504 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.157.196:36257, user_credentials={real_user=slave}} blocked reactor thread for 67035.5us
I20240503 08:13:43.489248 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb392d6697d1442d86d3a64f956e8211" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.489575 10700 raft_consensus.cc:3037] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:13:43.490993 10439 raft_consensus.cc:2396] T 6f8e6483973f420e97503f409130cc36 P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240503 08:13:43.491537 10569 raft_consensus.cc:482] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.491072 10168 catalog_manager.cc:5484] T f6443a176ba74f649ecc8da8717ed879 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 5 to 6, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 6 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.491705 10569 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.492018 10168 catalog_manager.cc:5484] T cb392d6697d1442d86d3a64f956e8211 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 4 to 6, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 6 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.492573 10168 catalog_manager.cc:5484] T 22b825c0acaa453f837aa7ca7a5543b0 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 6 to 7, leader changed from d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 7 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.494120 10697 raft_consensus.cc:1261] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Refusing update from remote peer d7d2dd10f0d54ac08dfdb8d2a348044a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240503 08:13:43.494366 10569 raft_consensus.cc:504] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.495080 10569 leader_election.cc:288] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249)
I20240503 08:13:43.494972 12797 consensus_queue.cc:1035] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:43.495754 10569 raft_consensus.cc:1226] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Rejecting Update request from peer d7d2dd10f0d54ac08dfdb8d2a348044a for earlier term 6. Current term is 7. Ops: []
I20240503 08:13:43.496057 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.496282 10439 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.496544 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b411562acf4c46c3b395d2901cf76a5a" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.496789 10305 raft_consensus.cc:3037] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.498271 10439 raft_consensus.cc:2396] T b411562acf4c46c3b395d2901cf76a5a P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20240503 08:13:43.499258 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8e6483973f420e97503f409130cc36" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.499557 10308 raft_consensus.cc:3032] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.499665 10308 raft_consensus.cc:729] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.500766 10305 raft_consensus.cc:2454] T b411562acf4c46c3b395d2901cf76a5a P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.500737 10308 consensus_queue.cc:260] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } }
I20240503 08:13:43.501251 10308 raft_consensus.cc:3037] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.502146 10505 leader_election.cc:302] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:43.502619 10569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.502910 10569 raft_consensus.cc:482] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.503108 10569 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.503825 10308 raft_consensus.cc:2396] T 6f8e6483973f420e97503f409130cc36 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240503 08:13:43.504325 10505 leader_election.cc:302] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2; no voters: 2b3a6ff465524577a400de6af7e70f71, d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:43.504853 12946 raft_consensus.cc:2781] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.505033 12946 raft_consensus.cc:686] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.505388 12946 consensus_queue.cc:237] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.506457 10569 raft_consensus.cc:504] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.506712 12785 raft_consensus.cc:2726] T 6f8e6483973f420e97503f409130cc36 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:13:43.507519 10569 leader_election.cc:288] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:43.508821 10568 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54e6601a404c46248232c7193aacaecb"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.508841 10697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.509045 10697 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.509042 10568 raft_consensus.cc:482] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.509199 10568 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.512730 10308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9595f5c7afc4218a96ef7a18a8d2e68" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.512817 10700 raft_consensus.cc:2396] T cb392d6697d1442d86d3a64f956e8211 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 08:13:43.512929 10568 raft_consensus.cc:504] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.513060 10308 raft_consensus.cc:3032] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.513271 10308 raft_consensus.cc:729] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d7d2dd10f0d54ac08dfdb8d2a348044a, State: Running, Role: LEADER
I20240503 08:13:43.513692 10308 consensus_queue.cc:260] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.514237 10568 leader_election.cc:288] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 54534138f6814b809a25c03a46e5f931 (127.27.157.196:36257)
I20240503 08:13:43.514623 10700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "54534138f6814b809a25c03a46e5f931"
I20240503 08:13:43.514746 10439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e6601a404c46248232c7193aacaecb" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.514899 10700 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.514964 10439 raft_consensus.cc:3037] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.515271 10568 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30"
dest_uuid: "22588cae35464393b2d6adb27e3114b2"
from {username='slave'} at 127.0.0.1:49660
I20240503 08:13:43.515488 10568 raft_consensus.cc:482] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.515581 10568 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.516549 10697 raft_consensus.cc:2396] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240503 08:13:43.516777 10701 raft_consensus.cc:1226] T d9595f5c7afc4218a96ef7a18a8d2e68 P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Rejecting Update request from peer d7d2dd10f0d54ac08dfdb8d2a348044a for earlier term 6. Current term is 7. Ops: []
I20240503 08:13:43.517818 10439 raft_consensus.cc:2454] T 54e6601a404c46248232c7193aacaecb P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.518388 10568 raft_consensus.cc:504] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.518558 10507 leader_election.cc:302] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, 2b3a6ff465524577a400de6af7e70f71; no voters:
I20240503 08:13:43.518986 12852 raft_consensus.cc:2781] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.519688 10568 leader_election.cc:288] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:43.519851 12852 raft_consensus.cc:686] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.519840 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.520136 10438 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.520197 10569 raft_consensus.cc:1226] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Rejecting Update request from peer d7d2dd10f0d54ac08dfdb8d2a348044a for earlier term 6. Current term is 7. Ops: []
I20240503 08:13:43.520498 12852 consensus_queue.cc:237] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.523465 12919 consensus_queue.cc:1046] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 08:13:43.524263 10308 raft_consensus.cc:3037] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.525630 10438 raft_consensus.cc:2396] T f1dff69d2bf84a45bdd1cd443d833f30 P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:13:43.526517 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1dff69d2bf84a45bdd1cd443d833f30" candidate_uuid: "22588cae35464393b2d6adb27e3114b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.526877 10305 raft_consensus.cc:3037] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.529717 10308 raft_consensus.cc:2396] T d9595f5c7afc4218a96ef7a18a8d2e68 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240503 08:13:43.530740 10305 raft_consensus.cc:2454] T f1dff69d2bf84a45bdd1cd443d833f30 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22588cae35464393b2d6adb27e3114b2 in term 7.
I20240503 08:13:43.531291 10505 leader_election.cc:302] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a; no voters: 2b3a6ff465524577a400de6af7e70f71
I20240503 08:13:43.531622 10504 leader_election.cc:302] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22588cae35464393b2d6adb27e3114b2; no voters: 54534138f6814b809a25c03a46e5f931, d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:43.532028 12947 raft_consensus.cc:2781] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.532188 12947 raft_consensus.cc:686] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Leader. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.532485 12947 consensus_queue.cc:237] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.532745 12785 raft_consensus.cc:2726] T d9595f5c7afc4218a96ef7a18a8d2e68 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:13:43.534772 10700 raft_consensus.cc:2396] T 54e6601a404c46248232c7193aacaecb P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 22588cae35464393b2d6adb27e3114b2 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 08:13:43.541615 10700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:43.541877 10700 raft_consensus.cc:482] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.542022 10700 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:13:43.542354 10168 catalog_manager.cc:5484] T b411562acf4c46c3b395d2901cf76a5a P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 5 to 7, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 7 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:43.544910 10700 raft_consensus.cc:504] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.545746 10700 leader_election.cc:288] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 8 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:43.546341 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.546331 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00782a28ed184a198567f3bd7808a5bb" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:43.546538 10438 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:13:43.546548 10569 raft_consensus.cc:3032] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Stepping down as leader of term 7
I20240503 08:13:43.546655 10569 raft_consensus.cc:729] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.546938 10569 consensus_queue.cc:260] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.547641 10569 raft_consensus.cc:3037] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:13:43.549453 10438 raft_consensus.cc:2454] T 00782a28ed184a198567f3bd7808a5bb P 2b3a6ff465524577a400de6af7e70f71 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 8.
I20240503 08:13:43.549409 10168 catalog_manager.cc:5484] T 54e6601a404c46248232c7193aacaecb P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 5 to 7, leader changed from 54534138f6814b809a25c03a46e5f931 (127.27.157.196) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 7 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.549960 10637 leader_election.cc:302] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters:
I20240503 08:13:43.550310 12866 raft_consensus.cc:2781] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 8 FOLLOWER]: Leader election won for term 8
I20240503 08:13:43.550060 10168 catalog_manager.cc:5484] T f1dff69d2bf84a45bdd1cd443d833f30 P 22588cae35464393b2d6adb27e3114b2 reported cstate change: term changed from 5 to 7, leader changed from 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194) to 22588cae35464393b2d6adb27e3114b2 (127.27.157.195). New cstate: current_term: 7 leader_uuid: "22588cae35464393b2d6adb27e3114b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.550477 12866 raft_consensus.cc:686] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [term 8 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:43.550729 10569 raft_consensus.cc:2396] T 00782a28ed184a198567f3bd7808a5bb P 22588cae35464393b2d6adb27e3114b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240503 08:13:43.550765 12866 consensus_queue.cc:237] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } }
I20240503 08:13:43.553985 10700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:43.554224 10700 raft_consensus.cc:482] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.554347 10700 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.554347 10162 catalog_manager.cc:5484] T 00782a28ed184a198567f3bd7808a5bb P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 7 to 8, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 8 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } }
I20240503 08:13:43.556851 10700 raft_consensus.cc:504] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.557548 10700 leader_election.cc:288] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 7 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:43.558247 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:43.558456 10569 raft_consensus.cc:3032] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:13:43.558555 10569 raft_consensus.cc:729] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.558836 10569 consensus_queue.cc:260] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.559175 10569 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.559551 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e2083a71dc4e9eb48eaa379874328d" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.559787 10438 raft_consensus.cc:3037] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.561226 10569 raft_consensus.cc:2396] T 71e2083a71dc4e9eb48eaa379874328d P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240503 08:13:43.562587 10438 raft_consensus.cc:2454] T 71e2083a71dc4e9eb48eaa379874328d P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 7.
I20240503 08:13:43.562997 10700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:43.563112 10637 leader_election.cc:302] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b3a6ff465524577a400de6af7e70f71, 54534138f6814b809a25c03a46e5f931; no voters: 22588cae35464393b2d6adb27e3114b2
I20240503 08:13:43.563239 10700 raft_consensus.cc:482] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.563405 10700 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:13:43.563671 12047 raft_consensus.cc:2781] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 08:13:43.563834 12047 raft_consensus.cc:686] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [term 7 LEADER]: Becoming Leader. State: Replica: 54534138f6814b809a25c03a46e5f931, State: Running, Role: LEADER
I20240503 08:13:43.564085 12047 consensus_queue.cc:237] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.566391 10700 raft_consensus.cc:504] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.567135 10700 leader_election.cc:288] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 8 election: Requested vote from peers 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929), d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359)
I20240503 08:13:43.567584 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:43.567087 10162 catalog_manager.cc:5484] T 71e2083a71dc4e9eb48eaa379874328d P 54534138f6814b809a25c03a46e5f931 reported cstate change: term changed from 6 to 7, leader changed from 22588cae35464393b2d6adb27e3114b2 (127.27.157.195) to 54534138f6814b809a25c03a46e5f931 (127.27.157.196). New cstate: current_term: 7 leader_uuid: "54534138f6814b809a25c03a46e5f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } health_report { overall_health: HEALTHY } } }
I20240503 08:13:43.567858 10569 raft_consensus.cc:3032] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Stepping down as leader of term 7
I20240503 08:13:43.567975 10569 raft_consensus.cc:729] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.568040 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a927d070284ea09cfc9884e2c4e2b7" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"
I20240503 08:13:43.568240 10305 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:13:43.568253 10569 consensus_queue.cc:260] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } } peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } }
I20240503 08:13:43.568598 10569 raft_consensus.cc:3037] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:13:43.571223 10305 raft_consensus.cc:2396] T 74a927d070284ea09cfc9884e2c4e2b7 P d7d2dd10f0d54ac08dfdb8d2a348044a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240503 08:13:43.571367 10569 raft_consensus.cc:2396] T 74a927d070284ea09cfc9884e2c4e2b7 P 22588cae35464393b2d6adb27e3114b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 54534138f6814b809a25c03a46e5f931 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20240503 08:13:43.571635 10700 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4670994e75a24234913724bac4c30ec4"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:43.571945 10700 raft_consensus.cc:482] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.572103 10700 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.572548 10635 leader_election.cc:302] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54534138f6814b809a25c03a46e5f931; no voters: 22588cae35464393b2d6adb27e3114b2, d7d2dd10f0d54ac08dfdb8d2a348044a
I20240503 08:13:43.573021 12866 raft_consensus.cc:2726] T 74a927d070284ea09cfc9884e2c4e2b7 P 54534138f6814b809a25c03a46e5f931 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240503 08:13:43.574743 10700 raft_consensus.cc:504] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b3a6ff465524577a400de6af7e70f71" member_type: VOTER last_known_addr { host: "127.27.157.194" port: 39249 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.575493 10700 leader_election.cc:288] T 4670994e75a24234913724bac4c30ec4 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 7 election: Requested vote from peers 2b3a6ff465524577a400de6af7e70f71 (127.27.157.194:39249), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:43.575999 10438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "2b3a6ff465524577a400de6af7e70f71"
I20240503 08:13:43.576198 10438 raft_consensus.cc:3037] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:13:43.576474 10697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:43.576663 10697 raft_consensus.cc:482] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.576776 10697 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:13:43.578651 10697 raft_consensus.cc:504] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.579231 10697 leader_election.cc:288] T d44e068b52864dde9d988775249cb8d2 P 54534138f6814b809a25c03a46e5f931 [CANDIDATE]: Term 8 election: Requested vote from peers d7d2dd10f0d54ac08dfdb8d2a348044a (127.27.157.193:38359), 22588cae35464393b2d6adb27e3114b2 (127.27.157.195:44929)
I20240503 08:13:43.579888 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:43.580098 10568 raft_consensus.cc:3032] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Stepping down as leader of term 7
I20240503 08:13:43.580190 10568 raft_consensus.cc:729] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 22588cae35464393b2d6adb27e3114b2, State: Running, Role: LEADER
I20240503 08:13:43.580247 10697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4764c2955533410188d40d9465156bcc"
dest_uuid: "54534138f6814b809a25c03a46e5f931"
from {username='slave'} at 127.0.0.1:46350
I20240503 08:13:43.580439 10697 raft_consensus.cc:482] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:13:43.580569 10697 raft_consensus.cc:3037] T 4764c2955533410188d40d9465156bcc P 54534138f6814b809a25c03a46e5f931 [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:13:43.580448 10568 consensus_queue.cc:260] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a" member_type: VOTER last_known_addr { host: "127.27.157.193" port: 38359 } } peers { permanent_uuid: "22588cae35464393b2d6adb27e3114b2" member_type: VOTER last_known_addr { host: "127.27.157.195" port: 44929 } } peers { permanent_uuid: "54534138f6814b809a25c03a46e5f931" member_type: VOTER last_known_addr { host: "127.27.157.196" port: 36257 } }
I20240503 08:13:43.580797 10568 raft_consensus.cc:3037] T d44e068b52864dde9d988775249cb8d2 P 22588cae35464393b2d6adb27e3114b2 [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:13:43.582173 10438 raft_consensus.cc:2454] T 4670994e75a24234913724bac4c30ec4 P 2b3a6ff465524577a400de6af7e70f71 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54534138f6814b809a25c03a46e5f931 in term 7.
I20240503 08:13:43.583611 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4670994e75a24234913724bac4c30ec4" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "22588cae35464393b2d6adb27e3114b2"
I20240503 08:13:43.583786 10305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44e068b52864dde9d988775249cb8d2" candidate_uuid: "54534138f6814b809a25c03a46e5f931" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d7d2dd10f0d54ac08dfdb8d2a348044a"